Order-aware Interactive Segmentation
Authors: Bin Wang
Anwesa Choudhuri
Meng Zheng
Zhongpai Gao
Benjamin Planche
Andong Deng
Qin Liu
Terrence Chen
Ulas Bagci
Ziyan Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Interactive Segmentation
Image Segmentation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose OIS: order-aware interactive segmentation, to explicitly integrate relative depth infomation into 2D interactive segmentation, which improves the model's ability to distinguish objects based on their relative depths from each other.

Abstract: Interactive segmentation aims to accurately segment target objects with minimal user interactions. However, current methods often fail to accurately separate target objects from the background, due to a limited understanding of order, the relative depth between objects in a scene. To address this issue, we propose OIS: order-aware interactive segmentation, where we explicitly encode the relative depth between objects into order maps. We introduce a novel order-aware attention, where the order maps seamlessly guide the user interactions (in the form of clicks) to attend to the image features. We further present an object-aware attention module to incorporate a strong object-level understanding to better differentiate objects with similar order.  Our approach allows both dense and sparse integration of user clicks, enhancing both accuracy and efficiency as compared to prior works.  Experimental results demonstrate that OIS achieves state-of-the-art performance, improving mIoU after one click by 7.61 on the HQSeg44K dataset and 1.32 on the DAVIS dataset as compared to the previous state-of-the-art SegNext, while also doubling inference speed compared to current leading methods.

Published as a conference paper at ICLR 2025
ORDER-AWARE INTERACTIVE SEGMENTATION
Bin Wang1.2 Anwesa Choudhuri', Meng Zheng", Zhongpai Gao²; Benjamin Planche", Andong Deng', Qin Liu", Terrence Chen", Ulas Bagci', Ziyan Wu?
WNrrthwestern University, Chicago, IL, USA
United Imaging Intelligence, Boston, MA, USA Unniversity of Central Florida, Orlando, FL, USA 'University of North Carolina at Chapel Hill, Chapel Hill, NC. USA [first. last Jenorthwesterne edu, {first 1ast Quui aoii-aii.om
ABSTRACT
Interactive segmentation aims to accurately segment target objects with minimal user interactions. However, current methods often fail accurately separate target objects from the background, due to a limited understanding of order, the relative depth between objects in a scene. To address this issue, we propose OIS: order- aware interactive segmentation, where we explicitly encode the relative depth be- tween objects into order maps. We introduce a novel order-aware attention, where the order maps seamlessly guide the user interactions (in the form of clicks) attend to the image features. further present an object-aware attention module incorporate strong object-level understanding to better differentiate objects with similar order. Our approach allows both dense and sparse integration clicks, enhancing both accuracy and efficiency compared to prior works. Experimental results demonstrate that OIS achieves state-of-the-an performance, improving mIoU after one click by 7.61 on the HQSeg44K dataset and 1.32 the DAVIS dataset as compared to the previous state-of-the-art SegNext, while also doubling inference speed compared to current leading methods.
Image
Ground Truth
SegNext
HQ-SAM
Ours
10U-51.39
Order
Order
Object
Object
Object
P.> Prompt Integration Method
Dense
Sparse
Dense Sparse
Comparison of our method with current state-of-theeart methods, SegNext (Liu et al, 2024a) and HQ-SAM (Ke et al. 2024). using 5 clicks (red dots represent positive clicks and green dots represent negative clicks). Our method is able to better distinguish the target (the gate) from the background (the trees and the building) and achieve significantly higher interactoon-ove--nnio (loU). This highlights the effectiveness of contributions: (a) order (the relative depth between objects), (b) object awareness, and (c) the combination of dense and sparse prompt integration.
Figure 1:
INTRODUCTION
Interactive segmentation allows users to segment objects or regions using prompts, which can be in the form of clicks, scribbles, or bounding boxes. This task is indispensable for applications like image editing (Ling et al. 2021 Abdal et al. 2020). video object tracking (Cheng et al., 2024 Bekuzarov al. 2023), 3D scene understanding (uue et al., 2023 Choi al., 2024), and medical
"This work done during the internship of Bin Wang United Imaging Intelligence, Boston, MA.

---

Published as a conference paper at ICLR 2025
image annotation (Ma al. 2024). The goal of this task is to minimize the number of user inter- actions without compromising on the segmentation accuracy. Achieving this requires the model to have a strong discriminative capability to retain only the foreground features relevant to the object of interest while discarding unrelated background regions, as guided by the user prompts.
Recent methods for interactive segmentation RITM (Sofiuuk et al. 2022). SimpleClick (Liu al. 2023). SegNext (Liu et al. 2024a). SAM (Kirillov et al. 2023). ). HQ-SAM (Ke et al. (024). Dy- naMiTe (Rana al.) (023)) sufter from three main issues. Firstly and most importandly current methods lack a sense of relative depth between objects in a scene, which makes it difficult for them to distinguish objects accurately. For instance, as illustrated in Figure I current state-of-the-
interactive segmentation methods like SegNext (Liu et al. (024a) and HQ-SAM (Ke et al., (022)) fail to separate the target object (the gate) from the background (trees and buildings). Each object has specific order, or relative depth from each other, corresponding to its location in 3D> space (e.g., the gate is in front of the building and trees in Figure DT. Current methods fail to incorporate this information, often leading to inaccurate segmentation masks with many false positives and false negatives (Figure D As a result, users are required to perform multiple interactions, i.c., redun- dant positive clicks to capture the entire foreground and negative clicks to exclude the background. Secondly, the encoded positive and negative clicks in current methods interact with the entire image features, causing intermingling of the foreground and background, and preventing strong discrim- inative representation of the target object. This is problematic because, intuitively, negative clicks should attend only to the background regions, and positive clicks should only interact with the fore- ground regions, as investigated in a prior work (Cheng et al. (024)), to allow better discrimination between the target object and background. Thirdly, current methods are either computationally in- efficient (methods with dense integration of prompts like RITM (Sofiikk et al. 2022), SimpleClick (Liu et al. 2023), and SegNext (Liu et al.] (024at) or suffer from misalignment between the image features and prompts (methods with sparse integration of prompts like SAM (Kirillov et al., 2023).
HQ-SAM (Ke et al. (0044), and DynaMITe (Rana et al. 2023)).
To address the aforementioned issues, in this work, we propose novel method called order-aware interactive segmentation (OIS). To address the first issue discussed above, OIS incorporates order, i... relative depth between objects in scene. For this, we construct order maps by utilizing an additional depth map and calculating the relative depth between each point in the image and the user prompt's location. As illustrated in Figure 3] the order map effectively captures the relative depths between the target object, indicated by the user's 's prompt, and other regions in the image. To efficiently integrate order maps in our framework, we propose a novel order-aware atlention, which guides the model to focus more on regions close to the prompt's order and less on regions that are farther from the prompt order. This enables the model to better understand the target object's position in 3D space, improving its ability to distinguish the target from surrounding objects.
While order-aware attention helps differentiate target objects at different depths, struggles to sep- arate objects with similar depths. To address this and the second issue discussed above, we incor- porate the notion of objects through object-aware attention, a novel prompt-basen masked cross- attention mechanism. It ensures that positive and negative prompts are separated to align with their own corresponding regions. The foreground and background prompts are encoded into respective foreground and background embeddings that attend to different foreground and background regions. Note that explicit foreground-backgroundd attention has been used previously for the video object segmentation task (Cheng et al. 22244) but to the best of our knowledge, we are the first to use this explicit separation for interactive segmentation. Unlike Cheng et al. (2024), we encode the fore- ground and background clicks into foreground and background embeddings to introduce explicit notion of the target object and enable the network to distinguish different objects with similar depth.
To address the third issue, we combine both dense and sparse integration of the prompts, improving accuracy and efficiency. We retain the superior spatial alignment of dense integration methods, while being as computationally efficient as sparse integration methods.
In summary, the contributions of our proposed OIS are listed as follows:
We propose order-aware attention, which integrates order, or relative depths between ob- jects, into interactive segmentation, improving the model's ability to distinguish target ob- jects based their relative depths from one another.
2

---

Published as a conference paper at ICLR 2025
We introduce object-aware attention to incorporate a strong understanding of objects. This enables our model to better differentiate objects with similar order.
Our approach combines both dense and sparse integration of prompts. This novel design improves the alignment between the image and prompts while maintaining computational efficiency.
Experimental results show that our OIS delivers state-of-the -fhte--rt performance with low la- tency, improving 7.61 mloU after one click one click on HQSeg44K (Ke et al., 2024). 1.32 DAVIS (Perazzi et al., 2016), and 2 times faster than the current best method, SegNext (Liu et al. 2024a).
RELATED WORK
Interactive segmentation. Current methods for interactive segmentation can be categorized into two groups: dense fusion and sparse fusion. Dense fusion includes RITM (Sofiiuk et al. 2022).
FocalClick (Chen et al. 20222, SimpleClick (Liu et al. 2023), InterFormer (Huang et 2002), ) and
SegNext (Liu et al.) 2024a). While sparse fusion mainly involves DynaMITe (Rana et al. (023))
SAM (Kirillov et al.] 2023) HR-SAM (Huang al.] (024), and HQ-SAM (Ke et al. 2024). In
dense fusion, user prompts are represented as dense maps and aligned with the image using atten- tion mechanisms. However, this approach increases inference time significantly due the costly attention between spatial feature maps (image and prompt dense map features). In contrast, sparse fusion converts user prompts into sparse embeddings, which lowers computational costs during cross-attention with image features. However, this efficiency comes at the cost of reduced alignment precision between the prompts and the image. We combine the best of both types of fusion, which re- tains the spatial alignment capabilities of dense fusion while improving the efficiency through sparse fusion. More importantly, all these methods lack relative 3D information, which makes difficult to accurately distinguish objects in complex scenarios. To address this issue, we introduce order infor- mation and aware attention to incorporate 3D spatial context, enhancing the prompt's ability to separate targets more effectively. More related works can be found in Appendix A.3
Depth in segmentation tasks. Many image and video segmentation tasks have utilized depth as an additional modality better handle foreground-
separation, such instance seg- mentation (Wu et al. 2023), semantic segmentation (Zhang et al. 2023b Yin et al. 2023), panoptic segmentation (Gao et al. 2022), video object segmentation (Liu al. (024b)), video panoptic seg- mentation (uuan et al.) 2022). However, depth has hardly been explored for interactive segmentation. The only related work, MM-SAM (Wang al., 2022). adopts an extra encoder to integrate depth. However, they achieve poor performance in common interactive segmentation benchmarks due to the insufficient fusion between prompt and depth. this work, instead of directly inputting depth, we introduce order map that combines both depth and user prompt to further strengthen the model sense of relative depth between objects.
Object understanding in segmentation tasks.
Recent transformer-basee approaches like DETR (Carion et al. 2020), MaskFormer (Cheng et al. 2021), Mask2Former (Cheng et al. 2022). ViSTR
2021. HODOR (Athar et al. 2022), etc, introduce the concept of abstract object queries to represent objects in images and videos and seamlessly perform tasks like object detec- tion and segmentation. Cutie (Cheng al. 2024) further enhances the object representations for the video object segmentation task by having separate foreground and background regions that the object queries attend to, allowing the queries to distinguish between the foreground and background. How- ever, the aforementioned methods were not directly applicable to the interactive segmentation task. To address this issue, SAM (Kirillov et al., 2023) introduces a sense of objects into the interactive segmentation task by encoding positive and negative prompts into sparse embeddings, all of which together represent the target object. However, SAM fails to clearly distinguish between the fore- ground and background because both the positive and the negative embeddings attend to the same regions, unlike Cutie. To address this limitation of SAM, we introduce a foreground -orouudd-bacgoouudd separation into interactive segmentation, referred to object-aware attention. It enables the sparse embeddings to only attend to their corresponding regions based on their type (positive click embed- dings attend to the foreground and negative click embeddings attend to the background).
WWang al.
3

---

Published as a conference paper at ICLR 2025
Order Map Generation
Object-level Understanding
Order-lavel Understanding
Depth
Order Mapa
Embeiiing
Sparre Embeetiing
3
Roe.
Image Features
Object oware
Imsale
Dioss ottontion
Click
Meok Guidinnce
Pros, Mask
Main Framework
Orden- weweeeeerellandeeraenn underranndyy
orbbeddinga
Cross CrossAltentio Altention
Seft Cenneee C00oo00 Co0o
Nesl So..ing
Drops altin
Drden-aa Dederawna aona
Daconter
Prompt Clicks
Danse emborddnng
K,
Vi
Prev. Mask
Image
Order Map Generation
Mask
Figure 2: Overview of our proposed OIS framework. Our order maps are generated (blue box on top-left) to capture the relative depth of objects in scene, described in Sec. 3.2 The order maps selectively guide the sparse embeddings to attend to the image features in our novel order-aware attention module (highlighted in blue inside the order-level understanding block), as described in Sec. 3.2] The object-aware attention module (highlighted in green the object-level understanding
block) imparts strong discriminative notion of objects, as discussed in Sec. 3.3 We utilize both
sparse and dense integration of prompts (highlighted in red), as described in Sec. 3.4
METHODOLOGGY
3.1 OVERVIEW
The framework of proposed OIS is depicted in Figure 2 Given image with target object, the image features are first extracted using an image encoder. Following prior works (Huang et al.) 2023 Kirillov al. 2023), the image is encoded only image features are reused for subsequent interactive stages when new user clicks are added. Each new user click is encoded in two ways, including dense and sparse embeddings (Sec. 3.4.. Dense embeddings are added to the image features to enhance prompt alignment, while sparse embeddings interact with the image features through two types of attention mechanisms: a) our proposed order- aware attention (Sec. (.2) and b) object-aware attention (Sec. 3.3). This enables the model to contain a sense of relative depth and the notion objects. Finally, the enhanced features are passed through decoder to predict the target segmentation mask.
save computational cost. The same extracted
3.2 ORDER-LEVEL UNDERSTANDING
Order map. To make the model have sense of relative depth, we first introduce the concept of order, which represents the relative depth between objects in scene. We formulate this information as order map, defined as the relative distance in camera optical axis direction. It is generated by combining the depth map R I. ffW of the image with each interactive prompt. Let Q" indicate the set of positive prompts, and Ω2" be the set of negative prompts. For the ith positive prompt Pf Q?,, the order map M is defined as:
1 M
MP
R
Σ PEE
Rypb
(1)
where R, is the depth value at the coordinate of the positive prompt, and M is the number of positive prompts. Notice that since all the positive prompts refer to the same object, they share a common order map, which means order map MP is the same for all positive prompts 1 QP.
4

---

Published as a conference paper at ICLR 2025
[3] Image
(b) Depth
(c) Crdar Arl"
(d) Order .vef
(e) Order MR
08
U4
02
Illustration of order map (after normalization into 0-1). Red dots in (c) indicate positive prompt clicks, and white dots in (d) and (e) represent negative prompt clicks. In order maps, darker means closer to prompt-selected object, while lighter areas farther the prompt-selected object.
Figure 3:
However, in contrast, each negative prompt can correspond to different objects in the background. so each of them has its own order map. For the ith negative prompt p." E _2", the corresponding order map Mï is expressed as:
= R Rp
(2)
where R2p is the depth value the coordinate of the negative prompt. In Figure it is difficult to distinguish the foreground sculpture frame from the background ball using the image alone. But with the help of order map, is easier to use prompt to separate them. Fjc the foreground frame is fully separated from the background, and in Figure d)), the background ball is also clearly distinguished with a negative prompt. This means that if false positives occur on the background ball, one negative prompt can now recognize the entire ball, reducing unnecessary interactions to remove false positives.
Order-aware attention. The order map, proposed above, guides the sparse embeddings attend to the image features via novel order-aware attention. In sparse embeddings S E N the first half of N embeddings are generated from the positive prompts and the second half from the negative prompts. We construct order mask M (0, 1) frW [MP, M" ...,....MW MT. Mš...., where the first half contains order maps shared by all positive prompts, and the second half consists of individual order maps for each negative prompt. In order-aware attention, the goal is to make the prompt aware of the order within the image. To achieve this, we update the sparse embeddings by performing masked cross-attention with the image features, where the order mask guides the attention to relevant regions. The order-aware attention is defined as:
S' = softmax(QK" - aM)V + +S,
where Q is the linear transformation of sparse embedding S, and K, V are the linear transformation of image features Inspired by Athar al. (2022). aM E (0,+00) is added control the attention with a learnable scale parameter a. Specifically, if region is close to the prompt-selected order (i.e., its corresponding order mask value near 0), it behaves like normal cross attention. However, if the region is far from the prompt-selected order (i.e., its order mask value approaches too after scaling), the attention weight approaches due to softmax, indicating minimal attention. In summary, regions close to the prompt-selected order receive higher attention, while far regions receive less.
3.3 OBJECT-LEVEL UNDERSTANDING
Although lrder-level understanding can guide the prompt in distinguishing objects from different orders, it faces the limitation of differentiating between objects within the same order. To solve this problem, we introduce object-aware attention to impart a sense of objects into our model.
Object-aware attention. After going through order-aware attention, the sparse embeddings (Sec. brrkkuund-bakkrouudd separated masked cross-attention following Cutie (Cheng et al., 2024). We call this our object-aware attention module. Specifically, in this module, the sparse embeddings generated from the positive clicks attend only to the fore- ground regions, while the sparse embeddings generated from the negative clicks focus only on the background regions. The foreground and background regions are obtained from the prediction gen- erated using the previous clicks. This explicit wwngground-bcckrroone notion of the target object into the sparse embeddings. Different from Cutie (Cheng et al. 2024). where multiple object queries are randomly initialized, we encode the foreground and background clicks to generate sparse embeddings, which act attributes of one single target object.
3.2 attend to the image features using
separation introduces a strong
5

---

Published as a conference paper at ICLR 2025
Methods
Backbone
NoC90 | NoC95
1-mloU f
5-mloU
NoF95 4
RITM (Sofiiuk et al.
2022
HRNet32.00
10.01
14.58
36.03
77.72
910
FocalClick Chen ot
2022
ScgF-B3-S2gaa
8.12
12.63
62.89
84.63
835
FocalClick Chen et
2022
SegF-B3-S2gsr
10.74
61.92
85.45
649
SimpleClick (Liu
2023
ViT-B
7.47
12.39
65.54
85.11
797
InterFormer (Huang al.
2023
ViT-B1024
7.17
10.77
64.40
82.62
658
SAM (Kirillov al.
2023
ViT-B1024
7.46
12.42
45.08
86.16
811
EifficientSAM (iong et al.
2024
VIT-T
10.11
14,60
77.90
EifficientSAM (Xiong et al.
2024
ViT-S1024
8.84
13.18
79.01
MobileSAM (Zhang ot al. 2023a
ViT-T1024
8.70
13.83
53.20
81.98
951
HQ-SAM Ke et 2024
ViT-B 1024
10.79
42.38
HR-SAM Huang et 2024
VIT- 102-
5.42
9.27
91.81
HR-SAM++ (Huang
2024
ViT-B1024
5.32
9.18
91.84
SegNext Liu ct al. 2024a
ViT-B
5.32
9.42
81.79
91.75
583
Ours
VIT- B12B
3.95
7,50
89.40
93.78
485
EifficientSAM Xioog et al.
2024
ViT-T20us
9.47
13.13
74.20
EifficientSAM Xiion et al.
2024
ViT-S2049
8.27
11.97
74.91
HR-SAM Huang et 2024
ViT-B2048
4.37
7.86
93.34
HR-SAM+ Huang et
2024
VIT- WTT-B2n4s
4.20
7.79
93.32
Ours
ViT-B
3.47
6.63
89.57
94.45
398
Table 1: Performance comparison the HQSeg44K benchmark.
Formally, the object-aware attention is defined as:
S' =oftmaxx(Ox" + H)V + S,
(4)
where Q is the linear transformation of sparse embedding S, and K, V are the linear transformation of image features F. Here, Mask H is
(z, =1
H(x. v) -00
(5)
otherwise
We construct mask H E 10 NXW [H H1 ".,,..,,ii' His H.,,..], where the first half consists of the previous mask H/ (with indicating the foreground target and for the background), and the second half consists of the inverse previous mask H (with 1 indicating the background and 0 for the foreground target). Note that the first round of interaction, when no previous mask is available, replace object-aware attention with the standard cross- cross-attention proposed in (Vaswani 2017
3.4 PROMPT INTEGRATION VIA DENSE AND SPARSE FUSION
Unlike previous methods that encode prompts through either dense sparse fusion, leverage the strengths of both approaches while addressing their limitations. Dense fusion can preserve more spatial details from the prompt (Liu et al. 2024a). By adding the dense embeddings D I #xxxxxx to the image features F xx the prompts achieve better spatial alignment with the image. Typically, this process is followed by self-attention module (Liu et al. 2024a) to capture spa- tial relationships. However, direct attention between spatial features is computationally expensive. To address this issue, we remove the self-attention module and introduce the sparse embeddings S E R XC where N is the number of prompts. Sparse fusion is more computationally efficient (Cheng et al. 2024), as the attention applied between sparse embeddings and spatial features. Moreover, sparse fusion offers greater flexibility for performing attention operations. Leveraging this flexibility, we have developed order-aware and object-aware attention mechanisms that enable the model to focus specific regions based on the sequence of user interactions or the contextual understanding of objects within the image. Thus, our approach improves the alignment between the image and prompt through dense embeddings while maintaining computational efficiency through sparse fusion and simultaneously leverages its flexibility to make the prompt aware order and object understanding.

---

Published as a conference paper at ICLR 2025
Image
Ground Truth
HQ-SAM
SegNext
Ours
Figure Qualitative results on HQSeg44K. Green dots indicate the user's first clicks.
EXPERIMENTS
4.1
EXPERIMENT SETTINGS
Datasets and evaluation metries. We evaluate our method and comparison methods on two widely used benchmarks for interactive segmentation: HQSeg44K (Ke et al., 2024) and DAVIS (Perazzi et al., (016)). More details about these two datasets can be found in Appendix A.2 We re- port five evaluation metrics: number of clicks (NoC), mean interaction-over-union (mloU), number of failures (NoF), seconds per click (SPC), and SAT latency. The NoC metrics, including NoC90 and NoC95, represent the number of clicks required to reach mloU thresholds of 90% and 95%, respectively. The mloU metrics, such as 1-mloU and 5-mloU, denote the average IoU achieved after 1 or 5 consecutive clicks. NoF measures the number cases that require more than 20 clicks to reach 90% mloU. SPC evaluates time efficiency by calculating the average time taken per click, while SAT latency refers to the total latency for the Segment Anything Task (with a grid of 16x16 points). For comparison methods, we utilize their released model weights to compute the 1-mloU and NoF95 scores. For all other metrics, we directly reference the results as reported in their papers.
Implementation details. In our framework, we use a frozen ViT-Base encoder from DepthAny- V2ingV2 (Ranfil et al., (02T) to encode each image. Note that we re-use the same encoder for depth map generation, allowing us to keep our overall model lightweight (as shown in Table 3). The en- coder is kept frozen during training retain the depth features. We have 2 decoders initialized from DepthAnything V2 (Ranftl et al.) 2021): one is kept frozen to generate the depth map and the other one is trained generate the final segmentation. We formulate the dense embeddings (introduced in Sec. 3.44 by constructing a two-channel dense map (one for positive clicks and the other for nega- tive clicks). Then, we encode the dense map via a single convolution layer. For sparse embeddings, we represent each click as a C-dimensional embedding (C = 128) following SAM (Kirillov et al.) 2023). Each of these embeddings is the sum of the positional encoding of its coordinate and learn- able embedding based on its prompt type (positive or negative). We set the maximum number of clicks N as during training, with the first 24 for positive clicks and the remaining for negative. there are fewer than 48 clicks, we fill the remaining slots with an additional 'non-point' embedding. We use three same blocks of object-level and lrder-level understanding modules (Sec. 3.2 and Sec. 3.3 conduct prompt fusion. In these modules, the Rerd-Forward Network (FFN) is implemented as a 2-layer Multi-Layer Perceptron (MLP). Each attention module is followed by a LayerNorm, which normalizes the sparse embeddings.
7

---

Published as a conference paper at ICLR 2025
Methods
Backbone
NoC90 1 NoC95
1-mloU f
5-mloU t
NoF95 4
RITM (Sofiiuk et al. 2022
HRNet32.00
5.34
11.45
72.53
89.75
139
FocalClick Chen ot al.
2022
ScgF-B3-S2gse
5.17
11.42
76.28
90.82
155
FocalClick Chen et 2022
SegF-B33S22gr
4.90
10.40
76.35
91.22
123
SimpleClick (Liu
2023
ViT-Baus
5.06
10.37
72.90
90.73
107
InterFormer (Huang al.
2023
ViT-B1024
5.45
11.88
64.40
87.79
150
SAM (Kirillov al.
2023
ViT-B1024
5.14
10.74
48.66
90.95
154
MobileSAM (Zhang et al.
2023a
VIT-T
5.83
12.74
61.69
89.18
196
EifficientSAM (iion et al.
2024
117-T1024
7.37
14.28
85.26
EMfficientAAM (Xiong et
2024
VIT-S1024
6.37
12.26
87.55
HQ-SAM Ke et al. 2024
ViT-B 1024
5.26
10.00
45.75
91.77
136
HR-SAM Huang et
2024
VIT- 102-
4.82
11.86
91.34
HR-SAM+* (Huang
2024
ViT-B1024
5.02
11.64
91.25
SegNext Liu ct al. 2024a
ViT-B
4.43
10.73
85.97
91.87
123
Ours
VIT- 1024
3.80
8,59
87.29
92.76
114
EifficientSAM Xioog et al.
2024
ViT-T20us
8.00
14.37
84.10
EifficientSAM Xiion et al.
2024
ViT-S2049
6.86
12,49
85.17
HR-SAM Huang et al. 2024
ViT-B2048
4.22
8,83
92.63
HR-SAM+ Huang et
2024
VIT- WTT-B2uc 2048
4.12
8.72
92.73
Ours
ViT-Bauas
3.48
8.42
88.05
92.90
105
Table 2: Performance comparison on the DAVIS benchmark.
Training and evaluation. To have fair comparison, follow the prior works (Sofiiuk al. 2022 [iu et al.] 2023; Huang et al.) 2023 Liu et al.] (0244) by applying the same click sampling strategies for samples training and evaluation, and adopting normalized focal loss our loss function. Details of the click strategy are in Appendix [ The input image size is set to 1024. We use Adam optimizer train our model on HQSeg44K dataset for 15 epochs on two A100 GPUs.
4.2
EVALUATION ON HQSEG44K
We first evaluate HQSeg44K dataset (Ke al. 2024). As shown in Table our method signif- icantly outperforms other methods across all the metrics. In particular, our NoC90 score shows an improvement of more than 1 click, while our NoC95 score improves by approximately 2 clicks. Fur- thermore, our 1-mloU achieves a remarkable improvement around 8 points compared to SegNext. This indicates that with only one click, our model can already provide an accurate and good-quality segmentation mask with mloU of 89.40% compared to 81. 79% of SegNext.
In Figure 4 we present some qualitative results from the HQSeg44K dataset with one user click input. In the first example, we observe that SegNext is negatively impacted by the background because the bike and window have similar color, leading to parts of the background being mistakenly classified as the target object. In contrast, our model accurately segments the bike without any false positives from the background. The second example highlights this issue even more clearly. SegNext struggles to segment the racket strings, as it is confused by the humans behind the racket. However, our method effectively alleviates this problem, successfully segmenting the strings without interference. In the third case, SegNext includes false positives from both background railing and the wall in front of the table, while our model avoids these errors entirely. This suggests that our method has superior ability distinguish the spatial position of target objects in 3D space. Overall, these examples demonstrate that the introduction order information effectively reduces false positives. (We also provide multi-round click results in Appendix A.7 and challenging cases in Appendix A.6)
4.3
EVALUATION ON DAVIS
Table 2 provides the comparison of different methods on the DAVIS (Perazzi et al. 2016) dataset. Our method achieves the state-of-the-an performance in both NoC and mloU metrics. Specifically, model requires only 8.59 clicks to reach 95% mloU, compared to 10.73 clicks for SegNext, which is over 2 clicks improvement. This means that even in complex scenarios, our method can predict high-quality segmentation masks and generalize well. Figure further illustrates this point.
8

---

Published as a conference paper at ICLR 2025
Ground Truth
HQ-SAM
MM-SAM
SegNext
Ours
Figure 5: Qualitative result on DAVIS. Green dots indicate the user's first clicks.
As shown in Figure 5 SegNext is notably affected by background objects, image blur, occlusion, and complex background. In the first case, the rhinoceros is occluded by the tree and separated into two parts. SegNext is only able to segment the part of the rhinoceros where click is placed. Our approach, however, successfully segments the entire rhinoceros with just click, highlighting our order- and object-level understanding. In the second case, due to the fast movement of the car, the image is blurred, especially in the background and around the vehicle. SegNext incorrectly part of the prediction. In contrast, our method handles
segments the regions surrounding the the image blur effectively. In the third case, due to numerous background objects around the human arm, SegNext fails to fully segment the arm, and also mistakenly includes parts of the background as false positives. We also display the results for SAM-based methods, including HQ-SAM (Ke et al.) 2024) and MM-SAM (Xiao al., 2024). Notice that MM-SAM incorporates depth information via an additional depth encoder. However, despite this enhancement, it still performs similarly to SegNext and fails in these cases. Additional depth integration comparisons are Appendix A.4
4.4 EFFICIENCY ANALYSIS
Methods
Parameters (M)
SPC (ms) 1.
SAT Latency (s) 1.
NoC90
5-mloU
SimpleClick
96.46
55
81.3
7.47
85.11
HQ-SAM
94.81
10
5.1
6.49
89.85
SegNext 113.79
58
20.6
5.32
91.75
Ours
107.88
9.2
3.95
93.78
Efficiency and accuracy comparison of interactive segmentation methods. Our model achieves the best balance, offering low latency with superior segmentation accuracy.
Table 3:
Time efficiency is critical in interactive segmentation models, as users expect minimal latency to ensure smooth and responsive interactions. In Table 3 we compare our methods with leading ap- proaches in terms of efficiency (SPC and SAT Latency) and accuracy (NoC90 and 5-mloU), The SPC (ms) reflects the average speed per click, while SAT Latency (s) captures the overall time, accounting for image processing, depth model prediction, click processing, and segmentation.
SimpleClick is the slowest in terms of SPC, primarily because it rrceensdess the image after each new user click. This introduces significant time costs that increase its overall SAT latency. In contrast, our method encodes the image only once, regardless the number of subsequent clicks, similar to HQ-SAM and SegNext. As result, exhibit lower SPC and SAT Latency.
Among these, although HQ-SAM is slightly faster, its accuracy is significantly lower than our model. SegNext, while being slightly less accurate than ours, performs times slower than our method in terms of SPC and SAT latency. This discrepancy arises because our method computes cross attention between sparse queries and spatial image feature, replacing the heavy self-attention between high resolution spatial features used in SegNext. summary, our method achieves the optimal balance between efficiency and accuracy, offering low latency while achieving superior segmentation quality.

---

Published as a conference paper at ICLR 2025
Attention Weights
Attention Weights after Order-aware Attn
Image
Target Object
Depth Map
before Order awdee--wwareeee Attn
Figure 6: Visualization of attention weights before and after applying order- aware attention. Atten- tion weights initially spread across overlapping objects with the same semantics, after applying order-aware attention, they focus solely on the target object.
4.5 ABLATIONS
To better understand how each proposed module performs, we conduct the ablation experiments here. Table demonstrates the importance of each module in the proposed method by analyzing performance across three different kinds of metrics (NoC90, 5-mloU, and NoF95) DAVIS dataset.
Importance of order-aware atfention. The 2nd row in T shows the effect of removing the order-aware attention module. We observe significant increase in NoC90 by 1.04 and a decrease 5-mloU by 1.15. This indicates the model requires more user interaction with- out order information. This validates our claim that order-aware attention helps distinguish be- tween objects at different depth levels. This is also highlighted in Figure with two examples where similar objects are overlapping. We vi- sualize the attention weights before and after integrating order-aware attention. We observe that before integrating order-aware attention, the attention weights are distributed across both objects. After incorporating the order information, the attention weights focus solely on the target object. More visualizations of attention weight are in Appendix A.5
Methods
NoC90 1
5-mloU f
NoF95
Full
3.80
92.76
114
w/o order
4.84 (+1.04)
91.61 (-1.15)
w/o object
3.92 (+0.12)
92.52 (-0.24)
125
w/o sparse
5.18 (+1.38)
89.81 (-2.95)
167
w/o dense
4.63 (+0.83)
91.31 (-1.45)
Table Ablation experiments on DAVIS.
Importance of object-aware attention. The 3rd row Table 4 shows the effect removing the object-aware attention. The result shows performance decrease, indicating that object-aware attention indispensable.
Importance of sparse embeddings. The 4th row of Table shows the performance after completely removing the sparse embedding discussed in Sec. 3.4 This configuration automatically removes both the order-aware and object-aware attention. The prompt integration is solely via the addition of dense embeddings to the image embeddings (Sec. 3.4.. We observe a significant performance drop, demonstrating the critical role of the sparse fusion module effective prompt integration.
Importance of dense embeddings. The 5th row of Table 4 shows the effect of removing the dense embeddings (Sec. 3.4). The prompt is only represented by the sparse embeddings in this config- uration. This results in large performance decline, as dense fusion improves spatial alignment between the image and prompt, making prompt interactions more effective.
5 CONCLUSION
In this work, we have introduced Order-Aware Interactive Segmentation (OIS). which incorporates 3D spatial context through the concept order into 2D interactive segmentation. By leveraging pro- posed order-aware attention, the model can better distinguish objects based on their relative depths. We introduced object-aware attention to enhance our model's ability to differentiate objects within the same depth level. We integrated user clicks using both sparse and dense representations, im- proving segmentation accuracy and computational efficiency. Experimental results validated that OIS significantly improves segmentation accuracy and speed compared to prior methods.
10

---

Published as a conference paper at ICLR 2025
Ethics Statement. We have utilized publicly available datasets for training and evaluation, ensur- ing compliance with relevant data usage policies and privacy regulations. All datasets used are anonymized to protect individual privacy, and personally identifiable information is included in our model or results. The model designed for general-purper image segmentation tasks; how- ever, recognize that its deployment in specific applications may raise ethical considerations. We advocate for the responsible use of our technology to prevent misuse in contexts that could in- fringe on privacy other ethical standards. Besides, all methodologies and experiments have been conducted with integrity, adhering to established research ethics guidelines. We are committed to transparency and reproducibility, providing thorough documentation of our processes to facilitate independent verification and ethical scrutiny. Finally, we recognize the importance of ongoing di- alogue regarding the ethical implications of advancements in image segmentation technologies and encourage the research community to engage in continuous evaluation to uphold ethical standards in future developments.
Reproducibility Statement. To ensure the reproducibility of our work, we extensively describe the implementation details, such as the pre-trained model, training data, and other important training hyperparameters in Sec 3 Sec and Sec A.I We will release all source code pending release
approval.

---

Published as a conference paper at ICLR 2025
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 8296-8305 2020.
Ali Athar, Jonathon Luiten, Alexander Hermans, Deva Ramanan, and Bastian Leibe. Hodor: High- level object descriptors for object re-segmentation in video learned from static images. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3022-
3031, 2022.
Maksym Bekuzarov, Ariana Bermudez, Joon- Youn-YYung Lee, and Hao Li. Xmem++: Production-level video segmentation from few annotated frames. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 635-644, 2023.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint 23.2::2302.12288 2023.
Yuri Y Boykov and M-P Jolly. Interactive graph cuts for optimal boundary & region segmentation of objects in nd images. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001, volume pp. 105-112. IEEE, 2001.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. Focalelick: Towards practical interactive image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1300-1309, 2022.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 1290-1299, 2022.
Ho Kei Cheng. Scoung Wug Oh, Brian Price, Joon- Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3151-3161 2024.
Seokhun Choi, Hyeonseop Song, Jaechul Kim, Taehyeong Kim, and Hoseok Do. Click-gaussian: Interactive segmentation to any 3d gaussians. arXiv preprint arXiv:2407. 1793, 2024.
Naiyu Gao, Fei He, Jian Jia, Yanhu Shan, Haoyang Zhang, Xin Zhao, and Kaiqi Huang. Panop- ticdepth: A unified framework for depth-aware panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1632-1642, 2022.
Leo Grady. Random walks for image segmentation.
IEEE transactions on pattern analysis and
machine intelligence, 28(11):1768- 17888,177,11118 2006.
Varun Gulshan, Carsten Rother, Antonio Criminisi, Andrew Blake, and Andrew Zisserman. Geodesic star convexity for interactive image segmentation. In 2010 IEEE Computer Society Conference on Computer Vision and Puttern Recognition, pp. 3129-3136, IEEE, 2010.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
12

---

Published as a conference paper at ICLR 2025
You Huang. Hao Yang, Ke Sun, Shengchuan Zhang, Liujuan Cao, Guannan Jiang, and Rongrong Ji. Interformer: Real-time interactive image segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22301-22311 2023.
You Huang, Wenbin Lai, Jiayi Ji, Liujuan Cao, Shengchuan Zhang, and Rongrong Ji. Hrsam: Efficiently segment anything in high-resolution images. arXiv preprint arXiv:2407.02100 2024.
Won-Dong Jang and Chang-Su Kim. Interactive image segmentation via backpropagating refine- ment scheme. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5297-5306, 2019.
Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu, al. Segment anything in high quality. Advances in Neural Information Processing Systems, 36, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 4015-4024 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Aision-ECCY 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Zheng Lin, Zhao Zhang, Lin-Zhuo Chen, Ming-Ming Cheng, and Shao-Ping Lu. Interactive image segmentation with first click attention. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13339-13348, 2020.
Huan Ling, Karsten Kreis, Daiqing Li, Scung Wook Kim, Antonio Torralba, and Sanja Fidler. Edit- gan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34:16331-1633 2021.
Qin Liu, Meng Zheng, Benjamin Planche, Srikrishna Karanam, Terrence Chen, Marc Niethammer, and Ziyan Wu. Pseudoclick: Interactive image segmentation with click imitation. In European Conference on Computer Vision, pp. 725-745. Springer, 2022.
Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22290-22300 2023.
Qin Liu, Jaemin Cho, Mohit Bansal, and Mare Niethammer. Rethinking interactive image seg- mentation with low latency high quality and diverse prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3773-3782 2024a.
Weihuang Liu, Xi Shen, Haolun Li, Xiuli Bi, Bo Liu, Chi-Man Pun, and Xiaodong Cun. Depth- aware test-time training for zero-shot video object segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2024b.
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
Kevis-Kokitsi Maninis, Sergi Caelles, Jordi Pont-Tuset, and Luc Van Gool. Deep extreme cut: From extreme points object segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 616-625, 2018.
Praful Mathur, Shashi Kumar Parwani, Mrinmoy Sen, Roopa Sheshadri, and Aman Sharma. Object aware contrastive prior for interactive image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 575-584, 2024.
Federico Perazzi, Jordi Pont-Tuset, Brian Mc McWilliams, Lue Van Gool, Markus Gross, and Alexander Sorkine-Hornung A benchmark dataset and evaluation methodology for video object segmen- tation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 724-732, 2016.
13

---

Published as a conference paper at ICLR 2025
Amit Kumar Rana, Sabarinath Mahadevan, Alexander Hermans, and Bastian Leibe. Dynamite: Dy- namic query bootstrapping for multi-object interactive segmentation transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 1043-1052, 2023.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference computer vision, pp. 12179-122 12188,
2021.
Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut" interactive foreground traction using iterated graph cuts. ACM transactions on graphics (TOG), 23(3):309-314, 2004.
Konstantin Sofiiuk, Ilia Petrov, Olga Barinova, and Anton Konushin. f-brs: Rethinking backpropa- gating refinemett for interactive segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8623- 8632, 2020.
Konstantin Sofiiuk, llya A Petrov, and Anton Konushin. Reviving iterative training with mask guid- ance for interactive segmentation. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 145. IEEE, 2022.
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12186-12195, 2022.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8741-8750, 2021.
Hallee E Wong, Marianne Rakic, John Guttag, and Adrian V Dalca. Scribbleprompt: Fast and flexible interactive segmentation for any medical image. arXiv preprint arXiv:23/2.07388/ 2023.
Zongwei Wu, Guillaume Allibert, Fabrice Meriaudeau, Chao Ma, and Cédric Demonccaux. Hidanet: Rgb-d salient object detection via hierarchical depth awareness. IEEE Transactions on Image Processing, 32:2160-2111 2173, 2023.
Aoran Xiao, Weihao Xuan, Heli Qi, Yun Xing, Naoto Yokoya, and Shijian Lu. Segment anything with multiple modalities. arXiv preprint arXiv:2408.0900 2024.
Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiao- liang Dai, Dilin Wang, Fei Sun, Forrest landola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16111-16121, 2024.
Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S Huang. Deep interactive object selection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 373-381, 2016.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10371-10381 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.00 09414, 2024b.
Bowen Yin, Xuying Zhang, Zhongyu Li, Li Liu, Ming-Ming Cheng, and Qibin Hou. Dformer: Rethinking rgbd representation learning for semantic segmentation. arXiv preprint arXiv:2309.09668. 2023.
Haobo Yuan, Xiangtai Li, Yibo Yang, Guangliang Cheng, Jing Zhang, Yunhai Tong, Lefei Zhang, and Dacheng Tao. Polyphonicformer: Unified query learning for depth-aware video panoptic segmentation. In European Conference on Computer Vision, pp. 582-599. Springer, 2022.
14

---

Published as a conference paper at ICLR 2025
Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, and Theodora Kontogianni. Agile3d: Attention guided interactive multi-object 3d segmentation. arXiv preprint arXiv:2306.00977 2023.
Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306. 14289, 2023a.
Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. IEEE Transactions intelligent transportation systems, 2023b.
Yian Zhao, Kehan Li, Zesen Cheng, Pengchong Qiao, Xiawu Zheng, Rongrong Ji, Chang Liu, Li Yuan, and Jie Chen. Graco: cranularity-controll interactive segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3501-3510, 2024.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 CLICK SAMPLING STRATEGY.
To have fair comparison, we apply the same click sampling strategy from prior works (Sofiiuk et al., 2022 Liu et al. 2023; Huang al. 2023 Liu et al. (0224a) for evaluation all the experiments. This strategy generates clicks sequentially, with each new click placed at the center of the largest error region the model's 's prediction.
A.2 MORE DATASETS DETAILS
We evaluate our method and comparison methods on two widely used benchmarks for interactive segmentation: HQSeg44K (Ke et al. (024) and DAVIS (Perazzi al. 2016). HQSeg44K is large- scale segmentation dataset containing 44320 images with high-quality mask labels. It contains a diverse range of images spanning over 1,000 semantic classes, covering both simple and complex scenarios, and includes objects with thin shapes as well as more straightforward forms. DAVIS high-precision video object segmentation dataset consisting of 50 videos. contains more complicated scenarios such occlusion, multi-objects, motion blurs and many other challenges. To be consistent with previous works, we use a subset of 345 frames to conduct the evaluation.
A.3 ADDITIONAL RELATED WORK ON INTERACTIVE SEGMENTATION
Early interactive segmentation methods (Boykov & Jolly 2001 Grady 2006 Rother et al. 2004
Gulshan al. 2010) relied on optimization techniques to solve cost functions defined by image pixels. As deep learning became more popular, approaches began incorporating user interactions directly into neural networks (Xu et al. 2016, Softiuk et al., 2020 Maninis et al. [018 Jang & Kim, 2019 Lin et al., 2020). Further advancements such as RITM (Sofiiuk et al. 2022, Liu et al., 2022 leveraged large-scale data for robustness, FocalClick (Chen et al. (022) introduced local re- finement modules, and SimpleClick (Liu et al. 2023) improved performance using vision transform- ers. OACE (Mathur et al. 2024) is proposed to improve the foreground distinction in a contrastive learning manner. To enhance efficiency, methods like InterFormer (Huang al. (023) reduced in- teraction time by encoding the image only once during interactions, and SegNext (Liu et al. 2024a)
applied similar idea with attention mechanism achieve high quality with low latency. Following the introduction of SAM (Kirillov et al., 2023), numerous SAM-based methods have been proposed to improve interactive segmentation in aspects such as quality (Ke et al. (024)). interaction effi- ciency (Zhang et al.) 2023a). and high-resolution image handling (Huang et al. (024). Variants of interactive segmentation, like multi-object segmentation (Yue et al.) [2023] Rana al. 2023), med- ical image segmentation (Ma et al., [2024] [Wong et al.) (023), and segmentation with controllable granularity (Zhao et al., 2024), have also been proposed.
A.4 COMPARISON WITH OTHER INTERACTIVE SEGMENTATION METHODS INCORPORATING 3D INFORMATION
In this section, we compare our methods other interactive segmentation methods that integrate 3D information (MM-SAM (Xiao et al. 2024 to show that naively incorporating depth is not as effective as our order-aware attention module discussed in Sec. 3.2 and validated Sec. 4 Here, we take the depth maps generated from DepthAnything V2 (Yang et al. (024a) as input to MM- SAM and test this on the DAVIS dataset; we call this configuration MM-SAM in Table Moreover, following the strategy of MM-SAM, which adds an extra encoder for other modalities such as lidar, depth, and thermal, we also train an additional Depth Encoder and add to our pipeline. Note that in this configuration, we disable the order-aware attention discussed in 3.2 to ensure fair comparison. We call this configuration DepthEncoder in Table 3 Our method outperforms this configuration and MM-SAM across all metrics. This indicates that incorporating 3D information into the model through order using our specialized order-aware attention module is a better solution than directly integrating depth maps.
16

---

Published as a conference paper at ICLR 2025
Methods
NoC90
NoC95
1-mloU 1
5-mloU 1
NoF95
MM-SAM (iiao et al. 2024)
6.22
12.26
51.41
88.19
181
Dephhnnender
3.88
10.19
85.36
92.15
140
3.80
8.59
92.76
Table Comparison with other interactive segmentation methods that integrate 3D information.
A.5 ADDITIONAL VISUALIZATION OF ATTENTION WEIGHTS IN ORDER-AWARE ATTENTION
In this section, we provide more visualizations of the attention weights show the importance of proposed order-aware attention, shown in Figure 7 and discussed in
Attention weights before Order-aware Attention
Attention weights After Order-aware Attention
Figure 7: Visualization of attention weights before and after applying order-aware attention.
Image Overlagped
Mask
Truth
Ours
Seggiout
Ours
Click lod--17
SingNest
Figure 8: Qualitative results for more challenging cases with multiple clicks. Green dots mean the positive clicks, and red dots are the negative clicks.
17

---

Published as a conference paper at ICLR 2025
A.6 QUALITATIVE RESULTS FOR MORE CHALLENGING CASES
We provide more qualitative results of challenging cases from the HQSeg44K (Ke et al. 2024 dataset displayed in Figure and Figure 9
Image
Ground Truth
Our Prob. Map
Our Pred. Mask
SegNext Prob. Map
SegNext Pred. Mask
Figure 9: Qualitative results for more challenging cases with only one click. The red star represents the first click.
A.7 ANALYSIS MULTI-ROUND INTERACTIONS
Interactive segmentation is multi-round task. The goal is to minimize the user interactions (i.e., interaction round number) and achieve a good quality segmentation mask. Hence, we evaluate the multi-round interaction performance of our method. Figure [T0 illustrates that while both our method and SegNext produce some false positives the background after the first click, our method eliminates the entire background, including regions inside the bike and adjacent to the human head, with only a single negative click. In contrast, SegNext requires 10 clicks to remove these background false positives and still multi-round interaction results in Figure [2 For the challenging cases, the multi-round interaction
get satisfied segmentation for the entire target. We also provide more
results are displayed in Figure I1 and Figure 13
Ours
Image
Click1 loU+42
Click2 loU=65
SegNext
Ground Truth
Click1 loU=28
Click2 loU=31
Click10 loU=56
Figure 10: Multi-round interaction comparison. Green dots mean the positive clicks, and red dots are the negative clicks.
18

---

Published as a conference paper at ICLR 2025
Ours
Ground Truth
SegNext
HO:SAM
Click2 loU 67.51
Click3 = 80.56
Figure 11: Multi-round interaction comparison for a challenging case.
Ours
SegNext
Ours
SegNext
Click 1 loUl-53.2
1oU-95.76
Click 1 loU-52.72
Click 2 1oL-80-45
1oU-03.15
CREC 3 10U-86.56
Click 1oU-94.34
1OU:9B.10
2
Click loU--9..00
Click 1oU-59.55
Figure 12: Multi-round interaction comparison. Green dots mean the positive clicks, and red dots the negative clicks.
19

---

Published as a conference paper at ICLR 2025
72.87
IoU
Image
Click
Click
Click 10
Click 20
Pred Mask
Figure 13: Multi-round interaction comparison for challenging case.
A.8 FAILURE CASES
In this section, discuss some failure cases of method. As shown in Figure T sometimes struggles to accurately predict the segmentation masks if thin objects occlude our target. This can be seen in the first example, where thin branches and leaves occlude the target bus and in the second and third examples, where the grass occludes the target human.
method
Image
with Pred. Mask Overlapped
Ground
Prob. Map
Pred. Mask
Dopth Map
Figure 14: Failure Cases. Green dots the first positive clicks.
A.9 IMPACT OF DEPTH MAP ON MODEL PERFORMANCE
To evaluate the impact of the quality of the depth map used three commonly used depth prediction models: DepthAnything V2 (Yang et al. 2024b), DepthAny- thing V1 (Yang et al. (2024a), and ZoeDepth (Bhat al. 2023). As illustrated in Fig. 15 DepthAny- thing V2 produces the most line-grained details, such as the girl's fingers and animal' 's tail, while the other methods generate lower-quality depth maps with some possible depth prediction errors.
model's performance, adopt
Note that these three models generate depth maps in different quality levels, allowing us to get better insights on our model's robustness to the depth quality.
Methods
Depth Model
NoC90
5-mloU 1
SegNext Tiu
2024a
4.43
91.87
OIS
DepthAnythingV2 Yang
2024b
3.80
92.76
OIS
DepthAnythingV1 Yang al. 2024a
3.78
92.69
OIS
ZoeDepth (Bat et al. 2023
3.75
92.75
Table 6: Performance comparison on DAVIS using depth maps from different depth prediction models.
20

---

Published as a conference paper at ICLR 2025
DepthAnything v2
DepthAnything v1
ZoeDepth
Depth Map
Prediction
Depth Map
Prediction
Figure 15: Qualitative results of model with different depth map generation models. Green dots are the first positive clicks.
We compared our model's performance on the DAVIS (Perazzi et al. (0166) dataset using depth maps from these three models. The results, presented Table 6, show minimal performance variation, with our method consistently outperforming the current SOTA method, SegNext (Liu et al.) 2024a). This is because the depth map is used solely to generate the order map, which guides the model's understanding of relative depth between objects. Even lower quality depth map (DepthAnything V1) has little impact on the final segmentation performance. More importantly, our proposed object- aware attention module is able negate the effects of erroneous order maps caused due erroncous depth maps.
Qualitative results provided in Fig. [5 It is observed that even though the qualities of the depth map are different, the prediction masks are nearly the same, especially the regions that depth maps have significant different (girls' hand and animal's tail), which indicates the robustness of model with different depth map source.
Moreover, it is important for the model remain robustness when meeting the depth prediction errors. In Fig. I6 we put two examples which contain depth prediction eтror. In the first case, the dancer's hat is incorrectly blending into the background audience in the predicted depth map. However, our model correctly recovers the error and accurately segments the hat. Additionally, in the second case, the duck's boundary closely mixes with the neighboring water terms of depth, while model successfully separates the duck from the water in the segmentation prediction. This robustness is attributed to our proposed object-aware understanding module which ensures that our model comprehends the object as whole, enabling it handle depth prediction errors effectively.
Besides, is noted that our model also achieve great performance in scenarios where significant depth variation exists within the target object. As demonstrated in Fig. 17 the target objects hibit considerable depth variation, yet our model consistently delivers accurate and high-quality segmentation. This stems from the proposed object-aware understanding module plays key role
21

---

Published as a conference paper at ICLR 2025
in ensuring the model perceives the target object whole, enabling it to handle significant depth variation effectively.
In summary, the key role of depth in approach is to address complex scenarios where distin- guishing the target from the background is challenging, as demonstrated Fig. [ and Fig. 4 In these scenarios, our model significantly outperforms existing methods. For cases where depth is less critical (Fig. [7). we show that our model remains robust and unaffected by potential negative effects of depth. This demonstrates that our model can effectively solve challenging cases while maintaining strong performance in standard scenarios, highlighting its superior performance and robustness.
DepthAnything v2
DepthAnything v1
ZoeDepth
Depth Map
Prediction
Depth Map
Prediction
Figure 16: Qualitative results of our model in scenarios with depth prediction errors or targets with depth similar to neighboring objects. Green dots are the first positive clicks.
A. ABLATION STUDY ABOUT THE ORDERING OF ORDER/ORJECT-AWWAEE UNDERSTANDING MODULES
Here, we discuss the effect of the sequence order of order-aware understanding and object-aware understanding module, We conduct the comparison experiment on DAVIS (Perazzi et al. 2016) dataset. The results in Table that having the order-aware understanding module first, followed by the object-aware understanding module slightly decreased performance than our current sequence (object-aware understanding module first, followed by the order-aware understanding module), in- dicating the effectiveness of our sequence choice.
A.11 ABLATION STUDY ON HQSEG44K DATASET
To further prove the importance of each our proposed module, here we conduct the ablation exper- iment HQSeg44K (Ke et al.) 2024) dataset. The results, displayed in Table 8 consistent with the findings from the ablation study on the DAVIS dataset, as shown in Table reaffirming that each proposed module plays crucial role in enhancing overall performance.
22

---

Published as a conference paper at ICLR 2025
Image overlapped with Pred. Mask
Depth Map
Ground Truth
Prob. Map
Pred. Mask
Figure 17: Qualitative results for cases with targets spanning a large depth range. Green dots are the first positive clicks.
Methods
NoC90 J
NoC95
1-mioU 1
5-mloU
order-aware understanding first
3.84
9.41
85.74
92.49
object-aware understanding first (current)
3.80
8.59
87.29
92.76
Table 7: Performance comparison on DAVIS with different sequence ordering of object and order- aware understanding modules.
Methods
NoC90
5-mloU t
Full
3,95
93.78
w/o order
4.87 (+0.98)
92.49 (-1.29)
w/o object
4.23 (+0.28)
93.28 (-0.5)
w/o sparse
5.23 (+1.28)
90.80 (-2.98)
w/o dense
4.97 (+1.02)
91.75 (-2.03)
Table 8: Ablation experiments on HQSeg44K.
23

---

Published as a conference paper at ICLR 2025
A.12 ABLATION STUDY FOR ORDER MAP WITH POSITIVE OR NEGATIVE CLICKS ALONE
We include additional ablation study on DAVIS (Perazzi et al.) 2016) dataset here analyze the design of the order map. Table | shows that removing either the positive click order map or the negative click order map leads to a performance drop, confirming the effectiveness of combining both. Interestingly, we observe the following:
Impact of Positive Click Order Map When only the negative click order map is used, the 1 1-mloU metric decreases more significantly. This suggests that the positive click order map particularly beneficial during the first click, as the first click is always a positive click.
Impact of Negative Click Order Map When only the positive click order map is used, the NoC and 5-mloU metrics see a larger decrease. This indicates that the negative click order map becomes more important as additional clicks are made. This is because subsequent clicks mainly involve adjustments and background removal, which rely heavily on negative clicks.
Methods
NoC90
NoC95 |
1-mloU
5-mloU f
pos+neg
3.80
8,59
87.29
92.76
pus
4.36
9.89
85.68
92.04
neg
4.01
9.24
84.17
92.37
Table 9: Ablation experiments for order with positive or negative clicks alone.
A. 3 ABLATION STUDY OF IMAGE ENCODER BACKBONE
To show the robustness of model different backbones, we conduct ablation study the HQSeg44K (Ke et al. 2024) dataset with different image encoder backbone. We replace the DepthAnything V2 backbone Yang al.) (024b) with an MAE pretrained VIT backbone (He al., 2022) to be consistent with prior SOTA methods, SegNext (Liu et al.] (024a), InterFormer (Huang al. 2023). and SimpleClick (Liu et al.) 2023). Table shows that our method still significantly outperforms the other methods with MAE VIT backbone. Furthermore, Table [T] highlights that the performance gains from our proposed order and object-aware attention far exceed those from switching backbones, representing the effectiveness and importance of our proposed methods.
Methods
Backbone
NoC90 L
NoC95 |
5-mloU 1
SimpleClick Liu et al. 2023
MAE ViT-B
7.47
12.39
85.11
InterFormer (Huang et al. 2023
MAE ViT-B
7.17
10.77
82.62
SegNext (Liu et al. 2024a
MAE ViT-B
5.32
9.42
91.75
OIS
MAE ViT-B
4.41
8.01
93.12
Table 10: Comparison of performance using the same backbone with other SOTA methods.
Methods
Backbone
NoC90
NoC95
5-mloU 1
OIS w/o order+object
MAE VIT-B
5.54
9.57
90.58
OIS
MAE VIT-B
4.41
8.01
93.12
OIS w/o onder+ohject
DepthAnythingV2 ViT-B
5.23
8.91
90.80
OIS
DepthAnythingV2 ViT-B
3.95
7.50
93.78
Table 11: Comparison of performance improvement of order and object-aware attention with the same backbone.
24

---

Published as a conference paper at ICLR 2025
A.14 PERFORMANCE COMPARISON TRAINED ON COCO
Since tradional interactive segmentation methods, including RITM (Sofiiuk et al. 2022). FocalClick
(Chen al. 2022), SimpleClick (Liu al. 2023). and InterFormer (Huang al. 2023 purely
trained on COCO+LVIS dataset (Li et al. 2014: Gupta et al. 2019). to have a fair comparison
with them, we also train our model using only COCO+LVIS dataset. In Table 12 provide
the comparison results on HQSeg44K (Ke et al., 2024 and DAVIS (Perazzi et al. 2016) dataset.
The results demonstrate that our method still outperforms other methods by large margin, which indicates the effectiveness of our model.
HQSeg44K NoC95
DAVIS NoC95
NoC90
1-mloU
NoC90
1-mloU f
RITM (Sofiiuk et al. 2022
10.01
14.58
36.03
5.34
11.45
72.53
FocalClick (Chen SimpleClick [Liu ct
2022
7.03
10.74
61.92
5.17
11.42
76.28
7.47
12.39
65.54
5.06
10.37
72.90
InterFormer (Huang ct al. 2023
7.17
10.77
64.40
5.45
11.88
64.40
OIS
5.16
9.18
85.36
4.41
9.87
87.21
Table 12: Performance comparison with methods trained on COCO+LVIS.
25


Reviews:
[{"review_id": "BNOjfyk5SH", "reviewer": "Reviewer_rqn7", "summary": "The paper presents Order-Aware Interactive Segmentation (OIS), combining order-aware and object-aware attention to improve segmentation accuracy and efficiency. Order maps help distinguish object depths, while foreground-background separation aids object differentiation. Using both dense and sparse prompt fusion, OIS achieves state-of-the-art results on HQSeg44K and DAVIS, boosting accuracy and speed.", "questions": "1. Would it be possible to show corresponding depth maps in Figure 6 to help readers better understand the importance of order-aware attention?\n2. As the number of negative clicks increases, does the model generate a unique order map for each negative prompt? If so, this approach could result in significant memory usage. Has the author considered selectively merging these maps? Since repeated negative clicks are likely to target very localized areas, the corresponding depth maps would likely have high similarity, making unique order maps for each negative prompt potentially redundant.\n3. Would lower-quality depth maps directly impact OIS performance, and conversely, would higher-quality maps improve it?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written, with clear motivation.\n2. It provides extensive experiments that convincingly demonstrate the proposed method's effectiveness.\n3. The visualized analysis adds value by highlighting the necessity of incorporating depth information.\n4. The method's performance surpasses current state-of-the-art methods.", "weaknesses": "1. The paper exceeds the ICLR 2025 10-page limit with 11 pages in the main text.\n2. Overall, the technical contribution and novelty of this paper are incremental, as it mainly incorporates existing priors, such as depth maps and foreground-background masks, to enhance segmentation accuracy. Since these priors have already proven effective in general segmentation tasks, their success in interactive segmentation is unsurprising. I would encourage the authors to clarify the unique benefits these priors bring specifically to interactive segmentation.\n3. The rationale for the ordering of Object-level Understanding before Order-level Understanding is unclear. Could the authors explain if this order enhances performance or if alternative orderings were tested?\n4. The modules for Object-level and Order-level Understanding are connected sequentially. Why was a parallel structure not considered? Could it improve performance or efficiency?\n5. Similar work (see [1]) also enhances foreground-background distinction in interactive segmentation; this should be discussed.\n6. The paper lacks ablation studies on Mask Guidance within the object-aware and order-aware attention modules. For example, testing the effects of different Mask Guidance qualities would be beneficial.\n7. It is unclear whether OIS can handle interactive segmentation for multiple objects simultaneously. Since Object-level Understanding relies on Mask Guidance from previous interactions, if the target object changes between clicks, OIS may struggle to shift focus to the new target due to constraints from the previous mask. This limitation could reduce the method's practical utility.\n\n[1] Object Aware Contrastive Prior for Interactive Image Segmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AJn0u83YM5", "reviewer": "Reviewer_r4kn", "summary": "The paper proposes order-aware interactive segmentation, which utilizes the extra relative depth information from pre-trained monocular depthanything v2 to generate the order maps. Then the order maps are used to guide the sparse embeddings to attend to the image features via mask attention. Object-aware attention is also used to help boost performance. Prompts are integrated via both the sparse and dense fusion. The paper validates its experiment design on HQSeg44K and DAVIS benchmark.", "questions": "What's the advantage of using depthanything v2's pretrained backbone in model's segmentation accuracy? besides saving parameters, how does it compare to the image encoder of SAM?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper has a good writing and structure, where the paper ideas and figures are easy to read and understand.\n\n2. The paper validates its model design choice in Sec 4.5 and Table 4, which shows the performance gain brought by each proposed component clearly.\n\n3. Using relative depth order to guide segmentation is an interesting idea, where the proposed order map considers both the positive and negative clicks.", "weaknesses": "1. The paper utilizes additional monocular depth prediction as input. In Table 3, does the computing cost of the depth model also get included? The paper should also study the impact of using various depth prediction models to the segmentation performance.\n\n2. The robustness of the segmentation model to the errors brought by depth prediction network is not studied. When depth prediction makes large errors, how will it influence the segmentation model? Especially, when segmenting neighboring objects with close depth distance, how much can the depth model contribute? \n\n3. Consider the importance of Table 4, besides DAVIS, an ablation experiment also on HQ-Seg44K should also be performed to have a more comprehensive understanding of each proposed component.\n\n4. The paper has a limited tech novelty, where the order map guiding attention is borrowed from Mask2Former and the object-aware attention is from Cuite. The depth prediction is from a pretrained depthanything v2 model. Using depth to guide more accurate segmentation is also introduced in [a, b]. This makes the paper more like a combination of existing components and model designs.\n\n[a] \"Depth-Guided Semi-Supervised Instance Segmentation.\" arXiv preprint arXiv:2406.17413\n[b] Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation And Sampling. CVPR, 2024.\n\n5. The paper misses an ablation experiment on order maps designs considering both for positive and negative clicks. What if only considering the order maps for positive or negative clicks alone?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DPtKGiQ3Zs", "reviewer": "Reviewer_QSAV", "summary": "1. This paper proposes order-aware attention, which integrates order (relative depths between objects) into interactive segmentation, improving the model’s ability to distinguish target objects based on their relative depths from one another. \n2. This paper introduces object-aware attention to incorporate a strong understanding of objects.\n3. This paper combines both dense and sparse integration of prompts, improving the alignment between the image and prompts while maintaining efficiency. \n4. This work achieves good performance with lower latency.", "questions": "1. Can the author connect and refine the motivation and the problem to be solved in this paper?\n2. See Weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The method looks very good for segmenting some difficult foreground objects (tennis rackets, bicycle wheels, etc.).\n2. The order-aware attention module introduced in this paper is easy to accept and effective.\n3. The framework of this paper is relatively concise and the implementation is easy to understand.", "weaknesses": "1. The method section of this paper is relatively easy to understand, but the motivation and problems to be solved are not concise enough. The author seems to want to solve many problems and propose many improvements, which can easily lead to readers not understanding why the author proposes certain techniques.\n2. The experiments of this paper is insufficient, lacking results on typical interactive segmentation datasets, such as GrabCut, Berkeley, SBD, PascalVOC, etc. In addition, there is a lack of comparison with some recently published (CVPR 2024, etc.) interactive segmentation methods.\n3. I admit the practical value of this paper, but I think it is more suitable to be published in CVPR rather than ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lXbupm2AOD", "reviewer": "Reviewer_jpbL", "summary": "This paper presents OIS, which explicitly encodes the relative depth between objects into order maps. It introduces an order-aware attention mechanism that guide the user interactions to attend to the image features by the order maps, and an object-aware attention module for better differentiation of objects with similar order. Experiments demonstrate the effectiveness and efficiency of OIS.", "questions": "The main motivation of this paper is to enhance the ability of interactive segmentation through depth information, but the paper seems to only show the cases where the depth of different positions of the same object is not very different, and there is little discussion about the effectiveness of the proposed method when the depth of different positions of the same object is very different (such as the bridge of DepthAnything visualization).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The OIS integrates relative depths between objects, into interactive segmentation, improving the model’s ability to distinguish target objects.\n2. Experimental results show quite good effectiveness.", "weaknesses": "1. The difference between order-aware attention and object-aware attention proposed in this paper and masked attention seems to be only in the source of input mask, so both are slightly innovative.\n2. In the comparative experiments of this paper, most of the traditional interactive segmentation methods (training on COCO) and SAM-based methods (training on SA-1B) have not been trained on HQSeg44K, while OIS is trained and tested on this dataset. The data quality of HQSeg44K is higher than that of COCO, so the fairness of the experiment is uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R86B12XcIc", "reviewer": "Reviewer_pFaJ", "summary": "Authors propose the order-aware interactive segmentation (OIS) to explicitly encode the relative depth between objects into order maps. Authors introduce a novel order-aware attention, where the order maps seamlessly guide the user interactions (in the form of clicks) to attend to the image features. Authors further present an object-aware attention module\nto incorporate a strong object-level understanding to better differentiate objects with similar order. OIS achieves state-of-the-art performance, improving mIoU after one click by 7.61 on the HQSeg44K dataset and 1.32 on the DAVIS dataset", "questions": "1. Using the same order map for all positive points may be too naive and in may condition, the depth values of different positive points maybe different.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tOIS can distinguish target objects based on their relative depths from one another and better differentiate objects with similar order.\n2.\tOIS improves the computational efficiency.\n3.\tOIS achieves competitive performance", "weaknesses": "1. The Concept of Order. Authors should reevaluate the current definition of 'order,' which is often misleading. The term 'order' should pertain to the sequence of click instead of depth map in interactive segmentation. \n2. Insufficient ablation study. More ablation studies are required to clarify the reason of performance improvement, for example the order of object-level attention and order-level attention, and the different pretrained weights of backbone, such as Depth Anything V1.\n3. Unfair comparison. OIS adopts Depth Anything V2 as backbone, while Depth Anything V2 requires much more (about 6x) pretraining images than SAM and other backbone. And MM-SAM (Table 5 in Appendix) with Depth Anything V2 achieves similar performance with OIS w/o order and object attention (Table 4). The comparison is unfair. Authors should conduct more experiments to further verify the effectiveness of order map. For example, authors can freeze the backbone of SAM and train an additional depth head to obtain the depth map. Then authors train the object and order attention to verify that the improvement comes from the order map instead of better pretrained backbone.\n3. Limited novelty. OIS integrates the object attention from Cutie and depth map into interactive segmentation. However, the novelty is somewhat constrained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2MMxEk8jq", "reviewer": "Reviewer_DpQL", "summary": "This work studies the interactive segmentation task. Based on the prior knowledge that ground and background objects are located at different depths, this work proposes a new framework named OIS. OIS effectively takes advantage of a corresponding depth map via the proposed order- and object-aware attention. Experiments demonstrate that OIS effectively improves segmentation performance by fewer clicks and boosts inference speed.", "questions": "Please refer to the WEAKNESSES part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript is clearly written and easy to follow. The whole framework is built upon the fact that different objects should be located at various depths in the original 3D scenes, which is promising. Both qualitative and quantitative comparisons demonstrate its superior performance.", "weaknesses": "Overall, I feel quite OK with this work, and there are no MAJOR weaknesses. See below for several suggestions and typos.\n\n- It is better to visualize the five clicks in Fig. 1 for an improved presentation.\n- Paragraph 2, Sec. 1, Page 1: Redundant '(' before 'RITM'.\n- Paragraph 2, Sec. 1, Page 2: Better to delete 'However' before 'current methods fail to ...'.\n- The blue dots in Fig. 3 are inconspicuous. Please consider another conspicuous color.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)