Order-aware Interactive Segmentation

## Abstract

Abstract Interactive segmentation aims to accurately segment target objects with minimal user interactions.
However, current methods often fail to accurately separate target objects from the background, due to a limited understanding of order , the relative depth between objects in a scene. To address this issue, we propose OIS: order-aware interactive segmentation, where we explicitly encode the relative depth between objects into order maps . We introduce a novel order-aware attention, where the order maps seamlessly guide the user interactions (in the form of clicks) to attend to the image features.
We further present an object-aware attention module to incorporate a strong object-level understanding to better differentiate objects with similar order.
Our approach allows both dense and sparse integration of user clicks, enhancing both accuracy and efficiency as compared to prior works.
Experimental results demonstrate that OIS achieves state-of-the-art performance, improving mIoU after one click by 7.61 7.61 7.61 7.61 on the HQSeg44K dataset and 1.32 1.32 1.32 1.32 on the DAVIS dataset as compared to the previous state-of-the-art SegNext, while also doubling inference speed compared to current leading methods. The project page is https://ukaukaaaa.github.io/OIS .

## 1 Introduction

Interactive segmentation allows users to segment objects or regions using prompts, which can be in the form of clicks, scribbles, or bounding boxes. This task is indispensable for applications like image editing (Ling et al., ; Abdal et al., ), video object tracking (Cheng et al., ; Bekuzarov et al., ), 3D scene understanding (Yue et al., ; Choi et al., ), and medical image annotation (Ma et al., ). The goal of this task is to minimize the number of user interactions without compromising on the segmentation accuracy. Achieving this requires the model to have a strong discriminative capability to retain only the foreground features relevant to the object of interest while discarding unrelated background regions, as guided by the user prompts.

Recent methods for interactive segmentation (RITM (Sofiiuk et al., ), SimpleClick (Liu et al., ), SegNext (Liu et al., ), SAM (Kirillov et al., ), HQ-SAM (Ke et al., ), DynaMITe (Rana et al., )) suffer from three main issues.
Firstly and most importantly, current methods lack a sense of relative depth between objects in a scene, which makes it difficult for them to distinguish objects accurately.
For instance, as illustrated in Figure 1, current state-of-the-art interactive segmentation methods like SegNext (Liu et al., ) and HQ-SAM (Ke et al., ) fail to separate the target object (the gate) from the background (trees and buildings).
Each object has a specific order, or relative depth from each other, corresponding to its location in 3D space (e.g., the gate is in front of the building and trees in Figure 1).
However, current methods fail to incorporate this information, often leading to inaccurate segmentation masks with many false positives and false negatives (Figure 1).
As a result, users are required to perform multiple interactions, i.e., redundant positive clicks to capture the entire foreground and negative clicks to exclude the background.
Secondly, the encoded positive and negative clicks in current methods interact with the entire image features, causing intermingling of the foreground and background, and preventing a strong discriminative representation of the target object. This is problematic because, intuitively, negative clicks should attend only to the background regions, and positive clicks should only interact with the foreground regions, as investigated in a prior work (Cheng et al., ), to allow better discrimination between the target object and background.
Thirdly, current methods are either computationally inefficient (methods with dense integration of prompts like RITM (Sofiiuk et al., ), SimpleClick (Liu et al., ), and SegNext (Liu et al., )) or suffer from misalignment between the image features and prompts (methods with sparse integration of prompts like SAM (Kirillov et al., ), HQ-SAM (Ke et al., ), and DynaMITe (Rana et al., )).

To address the aforementioned issues, in this work, we propose a novel method called order-aware interactive segmentation (OIS). To address the first issue discussed above, OIS incorporates order, i.e., relative depth between objects in a scene. For this, we construct order maps by utilizing an additional depth map and calculating the relative depth between each point in the image and the user prompt’s location. As illustrated in Figure 3, the order map effectively captures the relative depths between the target object, indicated by the user’s prompt, and other regions in the image.
To efficiently integrate order maps in our framework, we propose a novel order-aware attention, which guides the model to focus more on regions close to the prompt’s order and less on regions that are farther from the prompt order. This enables the model to better understand the target object’s position in 3D space, improving its ability to distinguish the target from surrounding objects.

While order-aware attention helps differentiate target objects at different depths, it struggles to separate objects with similar depths. To address this and the second issue discussed above, we incorporate the notion of objects through object-aware attention, a novel prompt-based masked cross-attention mechanism. It ensures that positive and negative prompts are separated to align with their own corresponding regions. The foreground and background prompts are encoded into respective foreground and background embeddings that attend to different foreground and background regions. Note that explicit foreground-background attention has been used previously for the video object segmentation task (Cheng et al., ), but to the best of our knowledge, we are the first to use this explicit separation for interactive segmentation. Unlike Cheng et al. (), we encode the foreground and background clicks into foreground and background embeddings to introduce an explicit notion of the target object and enable the network to distinguish different objects with similar depth.

To address the third issue, we combine both dense and sparse integration of the prompts, improving accuracy and efficiency.
We retain the superior spatial alignment of dense integration methods, while being as computationally efficient as sparse integration methods.

In summary, the contributions of our proposed OIS are listed as follows:

- •
We propose order-aware attention, which integrates order, or relative depths between objects, into interactive segmentation, improving the model’s ability to distinguish target objects based on their relative depths from one another.
- •
We introduce object-aware attention to incorporate a strong understanding of objects. This enables our model to better differentiate objects with similar order.
- •
Our approach combines both dense and sparse integration of prompts. This novel design improves the alignment between the image and prompts while maintaining computational efficiency.
- •
Experimental results show that our OIS delivers state-of-the-art performance with low latency, improving 7.61 mIoU after one click one click on HQSeg44K (Ke et al., ), 1.32 on DAVIS (Perazzi et al., ), and 2 times faster than the current best method, SegNext (Liu et al., ).

## 2 Related Work

#### Interactive segmentation.

Current methods for interactive segmentation can be categorized into two groups: dense fusion and sparse fusion. Dense fusion includes RITM (Sofiiuk et al., ), FocalClick (Chen et al., ), SimpleClick (Liu et al., ), InterFormer (Huang et al., ), and SegNext (Liu et al., ). While sparse fusion mainly involves DynaMITe (Rana et al., ), SAM (Kirillov et al., ), HR-SAM (Huang et al., ), and HQ-SAM (Ke et al., ).
In dense fusion, user prompts are represented as dense maps and aligned with the image using attention mechanisms. However, this approach increases inference time significantly due to the costly attention between spatial feature maps (image and prompt dense map features).
In contrast, sparse fusion converts user prompts into sparse embeddings, which lowers computational costs during cross-attention with image features. However, this efficiency comes at the cost of reduced alignment precision between the prompts and the image. We combine the best of both types of fusion, which retains the spatial alignment capabilities of dense fusion while improving the efficiency through sparse fusion. More importantly, all these methods lack relative 3D information, which makes it difficult to accurately distinguish objects in complex scenarios. To address this issue, we introduce order information and order-aware attention to incorporate 3D spatial context, enhancing the prompt’s ability to separate targets more effectively. More related works can be found in Appendix A.3.

#### Depth in segmentation tasks.

Many image and video segmentation tasks have utilized depth as an additional modality to better handle foreground-background separation, such as instance segmentation (Wu et al., ), semantic segmentation (Zhang et al., ; Yin et al., ), panoptic segmentation (Gao et al., ), video object segmentation (Liu et al., ), video panoptic segmentation (Yuan et al., ). However, depth has hardly been explored for interactive segmentation. The only related work, MM-SAM (Wang et al., ), adopts an extra encoder to integrate depth. However, they achieve poor performance in common interactive segmentation benchmarks due to the insufficient fusion between prompt and depth.
In this work, instead of directly inputting depth, we introduce order map that combines both depth and user prompt to further strengthen the model sense of relative depth between objects.

#### Object understanding in segmentation tasks.

Recent transformer-based approaches like DETR (Carion et al., ), MaskFormer (Cheng et al., ), Mask2Former (Cheng et al., ), ViSTR (Wang et al., ), HODOR (Athar et al., ), etc, introduce the concept of abstract object queries to represent objects in images and videos and seamlessly perform tasks like object detection and segmentation. Cutie (Cheng et al., ) further enhances the object representations for the video object segmentation task by having separate foreground and background regions that the object queries attend to, allowing the queries to distinguish between the foreground and background. However, the aforementioned methods were not directly applicable to the interactive segmentation task. To address this issue, SAM (Kirillov et al., ) introduces a sense of objects into the interactive segmentation task by encoding positive and negative prompts into sparse embeddings, all of which together represent the target object.
However, SAM fails to clearly distinguish between the foreground and background because both the positive and the negative embeddings attend to the same regions, unlike Cutie.
To address this limitation of SAM, we introduce a foreground-background separation into interactive segmentation, referred to as object-aware attention.
It enables the sparse embeddings to only attend to their corresponding regions based on their type (positive click embeddings attend to the foreground and negative click embeddings attend to the background).

## 3 Methodology

Figure: Figure 2: Overview of our proposed OIS framework. Our order maps are generated (blue box on the top-left) to capture the relative depth of objects in a scene, as described in Sec. 3.2. The order maps selectively guide the sparse embeddings to attend to the image features in our novel order-aware attention module (highlighted in blue inside the order-level understanding block), as described in Sec. 3.2. The object-aware attention module (highlighted in green in the object-level understanding block) imparts a strong discriminative notion of objects, as discussed in Sec. 3.3. We utilize both sparse and dense integration of prompts (highlighted in red), as described in Sec. 3.4.
Refer to caption: x2.png

### 3.1 Overview

The framework of our proposed OIS is depicted in Figure 2. Given an image with a target object, the image features are first extracted using an image encoder. Following prior works (Huang et al., ; Kirillov et al., ), the image is encoded only once to save computational cost. The same extracted image features are reused for subsequent interactive stages when new user clicks are added. Each new user click is encoded in two ways, including dense and sparse embeddings (Sec. 3.4). Dense embeddings are added to the image features to enhance prompt alignment, while sparse embeddings interact with the image features through two types of attention mechanisms: a) our proposed order-aware attention (Sec. 3.2) and b) object-aware attention (Sec. 3.3). This enables the model to contain a sense of relative depth and the notion of objects. Finally, the enhanced features are passed through a decoder to predict the target segmentation mask.

### 3.2 Order-level Understanding

#### Order map.

To make the model have a sense of relative depth, we first introduce the concept of order, which represents the relative depth between objects in a scene. We formulate this information as an order map, defined as the relative distance in camera optical axis direction.
It is generated by combining the depth map $\mathcal{R}\in\mathbb{R}^{H\times W}$ of the image with each interactive prompt. Let $\Omega^{p}$ indicate the set of positive prompts, and $\Omega^{n}$ be the set of negative prompts. For the $i$th positive prompt $p_{i}^{p}\in\Omega^{p}$, the order map $\mathcal{M}^{p}$ is defined as:

$$ $\mathcal{M}^{p}=\lvert\mathcal{R}-\frac{1}{M}\sum_{p_{i}^{p}\in\Omega^{p}}{ \mathcal{R}_{p_{i}^{p}}}\rvert,$ (1) $$

where $\mathcal{R}_{p_{i}^{p}}$ is the depth value at the coordinate of the positive prompt, and $M$ is the number of positive prompts. Notice that since all the positive prompts refer to the same object, they share a common order map, which means order map $\mathcal{M}^{p}$ is the same for all positive prompts $p_{i}^{p}\in\Omega^{p}$. However, in contrast, each negative prompt can correspond to different objects in the background, so each of them has its own order map. For the $i$th negative prompt $p_{i}^{n}\in\Omega^{n}$, the corresponding order map $\mathcal{M}_{i}^{n}$ is expressed as:

$$ $\mathcal{M}_{i}^{n}=\lvert\mathcal{R}-\mathcal{R}_{p_{i}^{n}}\rvert,$ (2) $$

where $\mathcal{R}_{p_{i}^{n}}$ is the depth value at the coordinate of the negative prompt. In Figure 3, it is difficult to distinguish the foreground sculpture frame from the background ball using the image alone. But with the help of order map, it is easier to use prompt to separate them. In Figure 3(c), the foreground frame is fully separated from the background, and in Figure 3(d), the background ball is also clearly distinguished with a negative prompt. This means that if false positives occur on the background ball, one negative prompt can now recognize the entire ball, reducing unnecessary interactions to remove false positives.

Figure: Figure 3: Illustration of order map (after normalization into 0-1). Red dots in (c) indicate positive prompt clicks, and blue dots in (d) and (e) represent negative prompt clicks. In order maps, darker means closer to prompt-selected object, while lighter areas are farther to the prompt-selected object.
Refer to caption: extracted/5933229/figs/order.png

#### Order-aware attention.

The order map, proposed above, guides the sparse embeddings to attend to the image features via our novel order-aware attention.
In sparse embeddings $\mathcal{S}\in\mathbb{R}^{N\times C}$, the first half of $N$ embeddings are generated from the positive prompts and the second half from the negative prompts. We construct order mask $\mathcal{M}\in(0,1)^{N\times HW}$ as $[\mathcal{M}^{p},\mathcal{M}^{p},...,\mathcal{M}_{1}^{n},\mathcal{M}_{2}^{n}...]$, where the first half contains order maps shared by all positive prompts, and the second half consists of individual order maps for each negative prompt. In order-aware attention, the goal is to make the prompt aware of the order within the image. To achieve this, we update the sparse embeddings by performing masked cross-attention with the image features, where the order mask guides the attention to relevant regions. The order-aware attention is defined as:

$$ $\mathcal{S}^{\prime}=\text{softmax}(QK^{T}-\sigma\mathcal{M})V+\mathcal{S},$ (3) $$

where $Q$ is the linear transformation of sparse embedding $\mathcal{S}$, and $K,V$ are the linear transformation of image features $\mathcal{F}$. Inspired by Athar et al. (), $\sigma\mathcal{M}\in(0,+\infty)$ is added to control the attention with a learnable scale parameter $\sigma$. Specifically, if a region is close to the prompt-selected order (i.e., its corresponding order mask value is near 0), it behaves like normal cross attention. However, if the region is far from the prompt-selected order (i.e., its order mask value approaches $+\infty$ after scaling), the attention weight approaches 0 due to softmax, indicating minimal attention. In summary, regions close to the prompt-selected order receive higher attention, while far regions receive less.

### 3.3 Object-level Understanding

Although order-level understanding can guide the prompt in distinguishing objects from different orders, it faces the limitation of differentiating between objects within the same order. To solve this problem, we introduce object-aware attention to impart a sense of objects into our model.

#### Object-aware attention.

After going through order-aware attention, the sparse embeddings (Sec. 3.2) attend to the image features using a foreground-background separated masked cross-attention following Cutie (Cheng et al., ). We call this our object-aware attention module. Specifically, in this module, the sparse embeddings generated from the positive clicks attend only to the foreground regions, while the sparse embeddings generated from the negative clicks focus only on the background regions. The foreground and background regions are obtained from the prediction generated using the previous clicks. This explicit foreground-background separation introduces a strong notion of the target object into the sparse embeddings.
Different from Cutie (Cheng et al., ), where multiple object queries are randomly initialized, we encode the foreground and background clicks to generate sparse embeddings, which act as attributes of one single target object.

Formally, the object-aware attention is defined as:

$$ $\mathcal{S}^{\prime}=\text{softmax}(QK^{T}+\mathcal{H})V+\mathcal{S},$ (4) $$

where $Q$ is the linear transformation of sparse embedding $\mathcal{S}$, and $K,V$ are the linear transformation of image features $\mathcal{F}$. Here, Mask $\mathcal{H}$ is

$$ $\mathcal{H}(x,y)=\begin{cases}0&\text{if }H(x,y)=1\\ -\infty&\text{otherwise}\end{cases}.$ (5) $$

We construct mask $H\in\{0,1\}^{N\times HW}$ as $[H^{f},H^{f},...,\hat{H^{f}},\hat{H^{f}},...]$, where the first half consists of the previous mask $H^{f}$ (with 1 indicating the foreground target and 0 for the background), and the second half consists of the inverse previous mask $\hat{H^{f}}$ (with 1 indicating the background and 0 for the foreground target).
Note that in the first round of interaction, when no previous mask is available, we replace object-aware attention with the standard cross-attention proposed in (Vaswani, ).

### 3.4 Prompt Integration via Dense and Sparse Fusion

Unlike previous methods that encode prompts through either dense or sparse fusion, we leverage the strengths of both approaches while addressing their limitations.
Dense fusion can preserve more spatial details from the prompt (Liu et al., ). By adding the dense embeddings $\mathcal{D}\in\mathbb{R}^{H\times W\times C}$ to the image features $\mathcal{F}\in\mathbb{R}^{H\times W\times C}$, the prompts achieve better spatial alignment with the image. Typically, this process is followed by a self-attention module (Liu et al., ) to capture spatial relationships. However, direct attention between spatial features is computationally expensive. To address this issue, we remove the self-attention module and introduce the sparse embeddings $\mathcal{S}\in\mathbb{R}^{N\times C}$, where $N$ is the number of prompts.
Sparse fusion is more computationally efficient (Cheng et al., ), as the attention is applied between sparse embeddings and spatial features.
Moreover, sparse fusion offers greater flexibility for performing attention operations.
Leveraging this flexibility, we have developed order-aware and object-aware attention mechanisms that enable the model to focus on specific regions based on the sequence of user interactions or the contextual understanding of objects within the image.
Thus, our approach improves the alignment between the image and prompt through dense embeddings while maintaining computational efficiency through sparse fusion and simultaneously leverages its flexibility to make the prompt aware of order and object understanding.

**Table 1: Performance comparison on the HQSeg44K benchmark.**
| Methods | Backbone | NoC90 $\downarrow$ | NoC95 $\downarrow$ | 1-mIoU $\uparrow$ | 5-mIoU $\uparrow$ | NoF95 $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| RITM (Sofiiuk et al., ) | $\text{HRNet32}_{400}$ | 10.01 | 14.58 | 36.03 | 77.72 | 910 |
| FocalClick (Chen et al., ) | $\text{SegF-B3-S2}_{256}$ | 8.12 | 12.63 | 62.89 | 84.63 | 835 |
| FocalClick (Chen et al., ) | $\text{SegF-B3-S2}_{384}$ | 7.03 | 10.74 | 61.92 | 85.45 | 649 |
| SimpleClick (Liu et al., ) | $\text{ViT-B}_{448}$ | 7.47 | 12.39 | 65.54 | 85.11 | 797 |
| InterFormer (Huang et al., ) | $\text{ViT-B}_{1024}$ | 7.17 | 10.77 | 64.40 | 82.62 | 658 |
| SAM (Kirillov et al., ) | $\text{ViT-B}_{1024}$ | 7.46 | 12.42 | 45.08 | 86.16 | 811 |
| EifficientSAM (Xiong et al., ) | $\text{ViT-T}_{1024}$ | 10.11 | 14.60 | - | 77.90 | - |
| EifficientSAM (Xiong et al., ) | $\text{ViT-S}_{1024}$ | 8.84 | 13.18 | - | 79.01 | - |
| MobileSAM (Zhang et al., ) | $\text{ViT-T}_{1024}$ | 8.70 | 13.83 | 53.20 | 81.98 | 951 |
| HQ-SAM (Ke et al., ) | $\text{ViT-B}_{1024}$ | 6.49 | 10.79 | 42.38 | 89.85 | 671 |
| HR-SAM (Huang et al., ) | $\text{ViT-B}_{1024}$ | 5.42 | 9.27 | - | 91.81 | - |
| HR-SAM++ (Huang et al., ) | $\text{ViT-B}_{1024}$ | 5.32 | 9.18 | - | 91.84 | - |
| SegNext (Liu et al., ) | $\text{ViT-B}_{1024}$ | 5.32 | 9.42 | 81.79 | 91.75 | 583 |
| Ours | $\text{ViT-B}_{1024}$ | 3.95 | 7.50 | 89.40 | 93.78 | 485 |
| EifficientSAM (Xiong et al., ) | $\text{ViT-T}_{2048}$ | 9.47 | 13.13 | - | 74.20 | - |
| EifficientSAM (Xiong et al., ) | $\text{ViT-S}_{2048}$ | 8.27 | 11.97 | - | 74.91 | - |
| HR-SAM (Huang et al., ) | $\text{ViT-B}_{2048}$ | 4.37 | 7.86 | - | 93.34 | - |
| HR-SAM++ (Huang et al., ) | $\text{ViT-B}_{2048}$ | 4.20 | 7.79 | - | 93.32 | - |
| Ours | $\text{ViT-B}_{2048}$ | 3.47 | 6.63 | 89.57 | 94.45 | 398 |

## 4 Experiments

Figure: Figure 4: Qualitative results on HQSeg44K. Green dots indicate the user’s first clicks.
Refer to caption: x3.png

### 4.1 Experiment Settings

#### Datasets and evaluation metrics.

We evaluate our method and comparison methods on two widely used benchmarks for interactive segmentation: HQSeg44K (Ke et al., ) and DAVIS (Perazzi et al., ). More details about these two datasets can be found in Appendix A.2.
We report five evaluation metrics: number of clicks (NoC), mean interaction-over-union (mIoU), number of failures (NoF), seconds per click (SPC), and SAT latency. The NoC metrics, including NoC90 and NoC95, represent the number of clicks required to reach mIoU thresholds of 90% and 95%, respectively. The mIoU metrics, such as 1-mIoU and 5-mIoU, denote the average IoU achieved after 1 or 5 consecutive clicks. NoF measures the number of cases that require more than 20 clicks to reach 90% mIoU. SPC evaluates time efficiency by calculating the average time taken per click, while SAT latency refers to the total latency for the Segment Anything Task (with a grid of 16×16 points). For comparison methods, we utilize their released model weights to compute the 1-mIoU and NoF95 scores. For all other metrics, we directly reference the results as reported in their papers.

#### Implementation details.

In our framework, we use a frozen ViT-Base encoder from DepthAnythingV2 (Ranftl et al., ) to encode each image. Note that we re-use the same encoder for depth map generation, allowing us to keep our overall model lightweight (as shown in Table 3). The encoder is kept frozen during training to retain the depth features. We have 2 decoders initialized from DepthAnythingV2 (Ranftl et al., ): one is kept frozen to generate the depth map and the other one is trained to generate the final segmentation.
We formulate the dense embeddings (introduced in Sec. 3.4) by constructing a two-channel dense map (one for positive clicks and the other for negative clicks). Then, we encode the dense map via a single convolution layer. For sparse embeddings, we represent each click as a $C$-dimensional embedding ($C=128$) following SAM (Kirillov et al., ). Each of these embeddings is the sum of the positional encoding of its coordinate and a learnable embedding based on its prompt type (positive or negative). We set the maximum number of clicks $N$ as 48 during training, with the first 24 for positive clicks and the remaining for negative. If there are fewer than 48 clicks, we fill the remaining slots with an additional ‘non-point’ embedding.
We use three same blocks of object-level and order-level understanding modules (Sec. 3.2 and Sec. 3.3) to conduct prompt fusion. In these modules, the Feed-Forward Network (FFN) is implemented as a 2-layer Multi-Layer Perceptron (MLP). Each attention module is followed by a LayerNorm, which normalizes the sparse embeddings.

#### Training and evaluation.

To have a fair comparison, we follow the prior works (Sofiiuk et al., ; Liu et al., ; Huang et al., ; Liu et al., ) by applying the same click sampling strategies for samples in training and evaluation, and adopting normalized focal loss as our loss function. Details of the click strategy are in Appendix A.1. The input image size is set to 1024. We use Adam optimizer to train our model on HQSeg44K dataset for 15 epochs on two A100 GPUs.

### 4.2 Evaluation on HQSeg44K

We first evaluate on HQSeg44K dataset (Ke et al., ). As shown in Table 1, our method significantly outperforms other methods across all the metrics.
In particular, our NoC90 score shows an improvement of more than 1 click, while our NoC95 score improves by approximately 2 clicks. Furthermore, our 1-mIoU achieves a remarkable improvement of around 8 points compared to SegNext. This indicates that with only one click, our model can already provide an accurate and good-quality segmentation mask with mIoU of 89.40% compared to 81.79% of SegNext.

In Figure 4, we present some qualitative results from the HQSeg44K dataset with one user click input. In the first example, we observe that SegNext is negatively impacted by the background because the bike and window have similar color, leading to parts of the background being mistakenly classified as the target object. In contrast, our model accurately segments the bike without any false positives from the background. The second example highlights this issue even more clearly. SegNext struggles to segment the racket strings, as it is confused by the humans behind the racket. However, our method effectively alleviates this problem, successfully segmenting the strings without interference. In the third case, SegNext includes false positives from both background railing and the wall in front of the table, while our model avoids these errors entirely. This suggests that our method has a superior ability to distinguish the spatial position of target objects in 3D space. Overall, these examples demonstrate that the introduction of order information effectively reduces false positives. (We also provide multi-round click results in Appendix A.7 and challenging cases in Appendix A.6.)

**Table 2: Performance comparison on the DAVIS benchmark.**
| Methods | Backbone | NoC90 $\downarrow$ | NoC95 $\downarrow$ | 1-mIoU $\uparrow$ | 5-mIoU $\uparrow$ | NoF95 $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| RITM (Sofiiuk et al., ) | $\text{HRNet32}_{400}$ | 5.34 | 11.45 | 72.53 | 89.75 | 139 |
| FocalClick (Chen et al., ) | $\text{SegF-B3-S2}_{256}$ | 5.17 | 11.42 | 76.28 | 90.82 | 155 |
| FocalClick (Chen et al., ) | $\text{SegF-B3-S2}_{384}$ | 4.90 | 10.40 | 76.35 | 91.22 | 123 |
| SimpleClick (Liu et al., ) | $\text{ViT-B}_{448}$ | 5.06 | 10.37 | 72.90 | 90.73 | 107 |
| InterFormer (Huang et al., ) | $\text{ViT-B}_{1024}$ | 5.45 | 11.88 | 64.40 | 87.79 | 150 |
| SAM (Kirillov et al., ) | $\text{ViT-B}_{1024}$ | 5.14 | 10.74 | 48.66 | 90.95 | 154 |
| MobileSAM (Zhang et al., ) | $\text{ViT-T}_{1024}$ | 5.83 | 12.74 | 61.69 | 89.18 | 196 |
| EifficientSAM (Xiong et al., ) | $\text{ViT-T}_{1024}$ | 7.37 | 14.28 | - | 85.26 | - |
| EifficientSAM (Xiong et al., ) | $\text{ViT-S}_{1024}$ | 6.37 | 12.26 | - | 87.55 | - |
| HQ-SAM (Ke et al., ) | $\text{ViT-B}_{1024}$ | 5.26 | 10.00 | 45.75 | 91.77 | 136 |
| HR-SAM (Huang et al., ) | $\text{ViT-B}_{1024}$ | 4.82 | 11.86 | - | 91.34 | - |
| HR-SAM++ (Huang et al., ) | $\text{ViT-B}_{1024}$ | 5.02 | 11.64 | - | 91.25 | - |
| SegNext (Liu et al., ) | $\text{ViT-B}_{1024}$ | 4.43 | 10.73 | 85.97 | 91.87 | 123 |
| Ours | $\text{ViT-B}_{1024}$ | 3.80 | 8.59 | 87.29 | 92.76 | 114 |
| EifficientSAM (Xiong et al., ) | $\text{ViT-T}_{2048}$ | 8.00 | 14.37 | - | 84.10 | - |
| EifficientSAM (Xiong et al., ) | $\text{ViT-S}_{2048}$ | 6.86 | 12.49 | - | 85.17 | - |
| HR-SAM (Huang et al., ) | $\text{ViT-B}_{2048}$ | 4.22 | 8.83 | - | 92.63 | - |
| HR-SAM++ (Huang et al., ) | $\text{ViT-B}_{2048}$ | 4.12 | 8.72 | - | 92.73 | - |
| Ours | $\text{ViT-B}_{2048}$ | 3.48 | 8.42 | 88.05 | 92.90 | 105 |

### 4.3 Evaluation on DAVIS

Table 2 provides the comparison of different methods on the DAVIS (Perazzi et al., ) dataset. Our method achieves the state-of-the-art performance in both NoC and mIoU metrics. Specifically, our model requires only 8.59 clicks to reach 95% mIoU, compared to 10.73 clicks for SegNext, which is over 2 clicks improvement. This means that even in complex scenarios, our method can predict high-quality segmentation masks and generalize well.
Figure 5 further illustrates this point. As shown in Figure 5, SegNext is notably affected by background objects, image blur, occlusion, and complex background. In the first case, the rhinoceros is occluded by the tree and separated into two parts. SegNext is only able to segment the part of the rhinoceros where a click is placed. Our approach, however, successfully segments the entire rhinoceros with just 1 click, highlighting our order- and object-level understanding. In the second case, due to the fast movement of the car, the image is blurred, especially in the background and around the vehicle. SegNext incorrectly segments the regions surrounding the car as part of the prediction. In contrast, our method handles the image blur effectively. In the third case, due to numerous background objects around the human arm, SegNext fails to fully segment the arm, and also mistakenly includes parts of the background as false positives. We also display the results for SAM-based methods, including HQ-SAM (Ke et al., ) and MM-SAM (Xiao et al., ). Notice that MM-SAM incorporates depth information via an additional depth encoder. However, despite this enhancement, it still performs similarly to SegNext and fails in these cases. Additional depth integration comparisons are in Appendix A.4.

Figure: Figure 5: Qualitative result on DAVIS. Green dots indicate the user’s first clicks.
Refer to caption: x4.png

### 4.4 Efficiency Analysis

**Table 3: Efficiency and accuracy comparison of interactive segmentation methods. Our model achieves the best balance, offering low latency with superior segmentation accuracy.**
| Methods | Parameters (M) | SPC (ms) ↓ | SAT Latency (s) ↓ | NoC90 ↓ | 5-mIoU ↑ |
| --- | --- | --- | --- | --- | --- |
| SimpleClick | 96.46 | 55 | 81.3 | 7.47 | 85.11 |
| HQ-SAM | 94.81 | 10 | 5.1 | 6.49 | 89.85 |
| SegNext | 113.79 | 58 | 20.6 | 5.32 | 91.75 |
| Ours | 107.88 | 31 | 9.2 | 3.95 | 93.78 |

Time efficiency is critical in interactive segmentation models, as users expect minimal latency to ensure smooth and responsive interactions. In Table 3, we compare our methods with leading approaches in terms of efficiency (SPC and SAT Latency) and accuracy (NoC90 and 5-mIoU). The SPC (ms) reflects the average speed per click, while SAT Latency (s) captures the overall time, accounting for both image processing and click processing.

SimpleClick is the slowest in terms of SPC, primarily because it re-encodes the image after each new user click. This introduces significant time costs that increase its overall SAT latency. In contrast, our method encodes the image only once, regardless of the number of subsequent clicks, similar to HQ-SAM and SegNext. As a result, we exhibit lower SPC and SAT Latency.

Among these, although HQ-SAM is slightly faster, its accuracy is significantly lower than our model. SegNext, while being slightly less accurate than ours, performs 2 times slower than our method in terms of SPC and SAT latency. This discrepancy arises because our method computes cross attention between sparse queries and spatial image feature, replacing the heavy self-attention between high resolution spatial features used in SegNext. In summary, our method achieves the optimal balance between efficiency and accuracy, offering low latency while achieving superior segmentation quality.

### 4.5 Ablations

Figure: Figure 6: Visualization of attention weights before and after applying order-aware attention. Attention weights initially spread across overlapping objects with the same semantics, but after applying order-aware attention, they focus solely on the target object.
Refer to caption: x5.png

To better understand how each proposed module performs, we conduct the ablation experiments here. Table 4 demonstrates the importance of each module in the proposed method by analyzing performance across three different kinds of metrics (NoC90, 5-mIoU, and NoF95) on DAVIS dataset.

**Table 4: Ablation experiments on DAVIS.**
| Methods | NoC90 ↓ | 5-mIoU ↑ | NoF95 ↓ |
| --- | --- | --- | --- |
| Full | 3.80 | 92.76 | 114 |
| w/o order | 4.84 (+1.04) | 91.61 (-1.15) | 171 |
| w/o object | 3.92 (+0.12) | 92.52 (-0.24) | 125 |
| w/o sparse | 5.18 (+1.38) | 89.81 (-2.95) | 167 |
| w/o dense | 4.63 (+0.83) | 91.31 (-1.45) | 188 |

Importance of order-aware attention. The 2nd row in Table 4 shows the effect of removing the order-aware attention module. We observe a significant increase in NoC90 by 1.04 and a decrease in 5-mIoU by 1.15. This indicates the model requires more user interaction without order information. This validates our claim that order-aware attention helps distinguish between objects at different depth levels. This is also highlighted in Figure 6 with two examples where similar objects are overlapping. We visualize the attention weights before and after integrating order-aware attention. We observe that before integrating order-aware attention, the attention weights are distributed across both objects. After incorporating the order information, the attention weights focus solely on the target object. More visualizations of attention weight are in Appendix A.5.

Importance of object-aware attention. The 3rd row in Table 4 shows the effect of removing the object-aware attention. The result shows a performance decrease, indicating that object-aware attention is indispensable.

Importance of sparse embeddings. The 4th row of Table 4 shows the performance after completely removing the sparse embedding discussed in Sec. 3.4. This configuration automatically removes both the order-aware and object-aware attention. The prompt integration is solely via the addition of dense embeddings to the image embeddings (Sec. 3.4). We observe a significant performance drop, demonstrating the critical role of the sparse fusion module in effective prompt integration.

Importance of dense embeddings. The 5th row of Table 4 shows the effect of removing the dense embeddings (Sec. 3.4). The prompt is only represented by the sparse embeddings in this configuration. This results in a large performance decline, as dense fusion improves spatial alignment between the image and prompt, making prompt interactions more effective.

## 5 Conclusion

In this work, we have introduced Order-Aware Interactive Segmentation (OIS), which incorporates 3D spatial context through the concept of order into 2D interactive segmentation. By leveraging proposed order-aware attention, the model can better distinguish objects based on their relative depths. We introduced object-aware attention to enhance our model’s ability to differentiate objects within the same depth level. We integrated user clicks using both sparse and dense representations, improving segmentation accuracy and computational efficiency. Experimental results validated that OIS significantly improves segmentation accuracy and speed as compared to prior methods.

## Appendix A Appendix

### A.1 Click Sampling Strategy.

To have fair comparison, we apply the same click sampling strategy from prior works (Sofiiuk et al., ; Liu et al., ; Huang et al., ; Liu et al., ) for evaluation in all the experiments. This strategy generates clicks sequentially, with each new click placed at the center of the largest error region in the model’s prediction.

### A.2 More Datasets Details

We evaluate our method and comparison methods on two widely used benchmarks for interactive segmentation: HQSeg44K (Ke et al., ) and DAVIS (Perazzi et al., ). HQSeg44K is a large-scale segmentation dataset containing 44320 images with high-quality mask labels. It contains a diverse range of images spanning over 1,000 semantic classes, covering both simple and complex scenarios, and includes objects with thin shapes as well as more straightforward forms. DAVIS is a high-precision video object segmentation dataset consisting of 50 videos. It contains more complicated scenarios such as occlusion, multi-objects, motion blurs and many other challenges. To be consistent with previous works, we use a subset of 345 frames to conduct the evaluation.

### A.3 Additional related work on interactive segmentation

Early interactive segmentation methods (Boykov & Jolly, ; Grady, ; Rother et al., ; Gulshan et al., ) relied on optimization techniques to solve cost functions defined by image pixels.
As deep learning became more popular, approaches began incorporating user interactions directly into neural networks (Xu et al., ; Sofiiuk et al., ; Maninis et al., ; Jang & Kim, ; Lin et al., ).
Further advancements such as RITM (Sofiiuk et al., ; Liu et al., ) leveraged large-scale data for robustness, FocalClick (Chen et al., ) introduced local refinement modules, and SimpleClick (Liu et al., ) improved performance using vision transformers. To enhance efficiency, methods like InterFormer (Huang et al., ) reduced interaction time by encoding the image only once during interactions, and SegNext (Liu et al., ) applied similar idea with attention mechanism to achieve high quality with low latency. Following the introduction of SAM (Kirillov et al., ), numerous SAM-based methods have been proposed to improve interactive segmentation in aspects such as quality (Ke et al., ), interaction efficiency (Zhang et al., ), and high-resolution image handling (Huang et al., ). Variants of interactive segmentation, like multi-object segmentation (Yue et al., ; Rana et al., ), medical image segmentation (Ma et al., ; Wong et al., ), and segmentation with controllable granularity (Zhao et al., ), have also been proposed.

### A.4 Comparison with Other 3D information Integration Method

In this section, we compare our methods to other 3D information integration methods to show that naively incorporating depth is not as effective as our order-aware attention module discussed in Sec. 3.2 and validated in Sec. 4. Following the strategy of MM-SAM (Xiao et al., ), which adds an extra encoder for other modalities such as lidar, depth, and thermal, we use the depth map generated by DepthAnything-V2 (Yang et al., ) into their additional encoder and test this configuration on the DAVIS dataset. Note that in this configuration, we disable the order-aware attention discussed in 3.2. We call this configuration DepthEncoder in Table 5, which shows the result of this setting. Our method outperforms this configuration and MM-SAM across all metrics. This indicates that formulating the 3D spatial context information into order using our specialized order-aware attention module is a better solution than directly integrating depth maps.

**Table 5: Comparison with other 3D information integration methods.**
| Methods | NoC90 ↓ | NoC95 ↓ | 1-mIoU ↑ | 5-mIoU ↑ | NoF95 ↓ |
| --- | --- | --- | --- | --- | --- |
| MM-SAM (Xiao et al., ) | 6.22 | 12.26 | 51.41 | 88.19 | 181 |
| DepthEncoder | 3.88 | 10.19 | 85.36 | 92.15 | 140 |
| Ours | 3.80 | 8.59 | 87.29 | 92.76 | 114 |

### A.5 Additional visualization of Attention Weights in Order-Aware Attention

In this section, we provide more visualizations of the attention weights to show the importance of proposed order-aware attention, as shown in Figure 7 and discussed in 4.5.

Figure: Figure 7: Visualization of attention weights before and after applying order-aware attention.
Refer to caption: x6.png

### A.6 Qualitative Results for More Challenging Cases

We provide more qualitative results of challenging cases from the HQSeg44K (Ke et al., ) dataset as displayed in Figure 8 and Figure 9.

Figure: Figure 8: Qualitative results for more challenging cases with multiple clicks. Green dots mean the positive clicks, and red dots are the negative clicks.
Refer to caption: x7.png

Figure: Figure 9: Qualitative results for more challenging cases with only one click. The red star represents the first click.
Refer to caption: x8.png

### A.7 Analysis of multi-round interactions

Interactive segmentation is a multi-round task. The goal is to minimize the user interactions (i.e., interaction round number) and achieve a good quality segmentation mask. Hence, we evaluate the multi-round interaction performance of our method. Figure 10 illustrates that while both our method and SegNext produce some false positives in the background after the first click, our method eliminates the entire background, including regions inside the bike and adjacent to the human head, with only a single negative click. In contrast, SegNext requires 10 clicks to remove these background false positives and still can not get satisfied segmentation for the entire target. We also provide more multi-round interaction results in Figure 12. For the challenging cases, the multi-round interaction results are displayed in Figure 11 and Figure 13.

Figure: Figure 10: Multi-round interaction comparison. Green dots mean the positive clicks, and red dots are the negative clicks.
Refer to caption: x9.png

Figure: Figure 11: Multi-round interaction comparison for a challenging case.
Refer to caption: x10.png

Figure: Figure 12: Multi-round interaction comparison. Green dots mean the positive clicks, and red dots are the negative clicks.
Refer to caption: x11.png

Figure: Figure 13: Multi-round interaction comparison for a challenging case.
Refer to caption: x12.png

### A.8 Failure Cases

In this section, we discuss some failure cases of our method. As shown in Figure 14, our method sometimes struggles to accurately predict the segmentation masks if thin objects occlude our target. This can be seen in the first example, where thin branches and leaves occlude the target bus and in the second and third examples, where the grass occludes the target human.

Figure: Figure 14: Failure Cases. Green dots are the first positive click.
Refer to caption: x13.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Would it be possible to show corresponding depth maps in Figure 6 to help readers better understand the importance of order-aware attention?
2. As the number of negative clicks increases, does the model generate a unique order map for each negative prompt? If so, this approach could result in significant memory usage. Has the author considered selectively merging these maps? Since repeated negative clicks are likely to target very localized areas, the corresponding depth maps would likely have high similarity, making unique order maps for each negative prompt potentially redundant.
3. Would lower-quality depth maps directly impact OIS performance, and conversely, would higher-quality maps improve it?
soundness: 3
strengths: 1. The paper is well-written, with clear motivation.
2. It provides extensive experiments that convincingly demonstrate the proposed method's effectiveness.
3. The visualized analysis adds value by highlighting the necessity of incorporating depth information.
4. The method's performance surpasses current state-of-the-art methods.
summary: The paper presents Order-Aware Interactive Segmentation (OIS), combining order-aware and object-aware attention to improve segmentation accuracy and efficiency. Order maps help distinguish object depths, while foreground-background separation aids object differentiation. Using both dense and sparse prompt fusion, OIS achieves state-of-the-art results on HQSeg44K and DAVIS, boosting accuracy and speed.
weaknesses: 1. The paper exceeds the ICLR 2025 10-page limit with 11 pages in the main text.
2. Overall, the technical contribution and novelty of this paper are incremental, as it mainly incorporates existing priors, such as depth maps and foreground-background masks, to enhance segmentation accuracy. Since these priors have already proven effective in general segmentation tasks, their success in interactive segmentation is unsurprising. I would encourage the authors to clarify the unique benefits these priors bring specifically to interactive segmentation.
3. The rationale for the ordering of Object-level Understanding before Order-level Understanding is unclear. Could the authors explain if this order enhances performance or if alternative orderings were tested?
4. The modules for Object-level and Order-level Understanding are connected sequentially. Why was a parallel structure not considered? Could it improve performance or efficiency?
5. Similar work (see [1]) also enhances foreground-background distinction in interactive segmentation; this should be discussed.
6. The paper lacks ablation studies on Mask Guidance within the object-aware and order-aware attention modules. For example, testing the effects of different Mask Guidance qualities would be beneficial.
7. It is unclear whether OIS can handle interactive segmentation for multiple objects simultaneously. Since Object-level Understanding relies on Mask Guidance from previous interactions, if the target object changes between clicks, OIS may struggle to shift focus to the new target due to constraints from the previous mask. This limitation could reduce the method's practical utility.

[1] Object Aware Contrastive Prior for Interactive Image Segmentation

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: What's the advantage of using depthanything v2's pretrained backbone in model's segmentation accuracy? besides saving parameters, how does it compare to the image encoder of SAM?
soundness: 2
strengths: 1. The paper has a good writing and structure, where the paper ideas and figures are easy to read and understand.

2. The paper validates its model design choice in Sec 4.5 and Table 4, which shows the performance gain brought by each proposed component clearly.

3. Using relative depth order to guide segmentation is an interesting idea, where the proposed order map considers both the positive and negative clicks.
summary: The paper proposes order-aware interactive segmentation, which utilizes the extra relative depth information from pre-trained monocular depthanything v2 to generate the order maps. Then the order maps are used to guide the sparse embeddings to attend to the image features via mask attention. Object-aware attention is also used to help boost performance. Prompts are integrated via both the sparse and dense fusion. The paper validates its experiment design on HQSeg44K and DAVIS benchmark.
weaknesses: 1. The paper utilizes additional monocular depth prediction as input. In Table 3, does the computing cost of the depth model also get included? The paper should also study the impact of using various depth prediction models to the segmentation performance.

2. The robustness of the segmentation model to the errors brought by depth prediction network is not studied. When depth prediction makes large errors, how will it influence the segmentation model? Especially, when segmenting neighboring objects with close depth distance, how much can the depth model contribute? 

3. Consider the importance of Table 4, besides DAVIS, an ablation experiment also on HQ-Seg44K should also be performed to have a more comprehensive understanding of each proposed component.

4. The paper has a limited tech novelty, where the order map guiding attention is borrowed from Mask2Former and the object-aware attention is from Cuite. The depth prediction is from a pretrained depthanything v2 model. Using depth to guide more accurate segmentation is also introduced in [a, b]. This makes the paper more like a combination of existing components and model designs.

[a] "Depth-Guided Semi-Supervised Instance Segmentation." arXiv preprint arXiv:2406.17413
[b] Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation And Sampling. CVPR, 2024.

5. The paper misses an ablation experiment on order maps designs considering both for positive and negative clicks. What if only considering the order maps for positive or negative clicks alone?

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Can the author connect and refine the motivation and the problem to be solved in this paper?
2. See Weaknesses section.
soundness: 4
strengths: 1. The method looks very good for segmenting some difficult foreground objects (tennis rackets, bicycle wheels, etc.).
2. The order-aware attention module introduced in this paper is easy to accept and effective.
3. The framework of this paper is relatively concise and the implementation is easy to understand.
summary: 1. This paper proposes order-aware attention, which integrates order (relative depths between objects) into interactive segmentation, improving the model’s ability to distinguish target objects based on their relative depths from one another. 
2. This paper introduces object-aware attention to incorporate a strong understanding of objects.
3. This paper combines both dense and sparse integration of prompts, improving the alignment between the image and prompts while maintaining efficiency. 
4. This work achieves good performance with lower latency.
weaknesses: 1. The method section of this paper is relatively easy to understand, but the motivation and problems to be solved are not concise enough. The author seems to want to solve many problems and propose many improvements, which can easily lead to readers not understanding why the author proposes certain techniques.
2. The experiments of this paper is insufficient, lacking results on typical interactive segmentation datasets, such as GrabCut, Berkeley, SBD, PascalVOC, etc. In addition, there is a lack of comparison with some recently published (CVPR 2024, etc.) interactive segmentation methods.
3. I admit the practical value of this paper, but I think it is more suitable to be published in CVPR rather than ICLR.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: The main motivation of this paper is to enhance the ability of interactive segmentation through depth information, but the paper seems to only show the cases where the depth of different positions of the same object is not very different, and there is little discussion about the effectiveness of the proposed method when the depth of different positions of the same object is very different (such as the bridge of DepthAnything visualization).
soundness: 3
strengths: 1. The OIS integrates relative depths between objects, into interactive segmentation, improving the model’s ability to distinguish target objects.
2. Experimental results show quite good effectiveness.
summary: This paper presents OIS, which explicitly encodes the relative depth between objects into order maps. It introduces an order-aware attention mechanism that guide the user interactions to attend to the image features by the order maps, and an object-aware attention module for better differentiation of objects with similar order. Experiments demonstrate the effectiveness and efficiency of OIS.
weaknesses: 1. The difference between order-aware attention and object-aware attention proposed in this paper and masked attention seems to be only in the source of input mask, so both are slightly innovative.
2. In the comparative experiments of this paper, most of the traditional interactive segmentation methods (training on COCO) and SAM-based methods (training on SA-1B) have not been trained on HQSeg44K, while OIS is trained and tested on this dataset. The data quality of HQSeg44K is higher than that of COCO, so the fairness of the experiment is uncertain.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Using the same order map for all positive points may be too naive and in may condition, the depth values of different positive points maybe different.
soundness: 3
strengths: 1.	OIS can distinguish target objects based on their relative depths from one another and better differentiate objects with similar order.
2.	OIS improves the computational efficiency.
3.	OIS achieves competitive performance
summary: Authors propose the order-aware interactive segmentation (OIS) to explicitly encode the relative depth between objects into order maps. Authors introduce a novel order-aware attention, where the order maps seamlessly guide the user interactions (in the form of clicks) to attend to the image features. Authors further present an object-aware attention module
to incorporate a strong object-level understanding to better differentiate objects with similar order. OIS achieves state-of-the-art performance, improving mIoU after one click by 7.61 on the HQSeg44K dataset and 1.32 on the DAVIS dataset
weaknesses: 1. The Concept of Order. Authors should reevaluate the current definition of 'order,' which is often misleading. The term 'order' should pertain to the sequence of click instead of depth map in interactive segmentation. 
2. Insufficient ablation study. More ablation studies are required to clarify the reason of performance improvement, for example the order of object-level attention and order-level attention, and the different pretrained weights of backbone, such as Depth Anything V1.
3. Unfair comparison. OIS adopts Depth Anything V2 as backbone, while Depth Anything V2 requires much more (about 6x) pretraining images than SAM and other backbone. And MM-SAM (Table 5 in Appendix) with Depth Anything V2 achieves similar performance with OIS w/o order and object attention (Table 4). The comparison is unfair. Authors should conduct more experiments to further verify the effectiveness of order map. For example, authors can freeze the backbone of SAM and train an additional depth head to obtain the depth map. Then authors train the object and order attention to verify that the improvement comes from the order map instead of better pretrained backbone.
3. Limited novelty. OIS integrates the object attention from Cutie and depth map into interactive segmentation. However, the novelty is somewhat constrained.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Please refer to the WEAKNESSES part.
soundness: 3
strengths: The manuscript is clearly written and easy to follow. The whole framework is built upon the fact that different objects should be located at various depths in the original 3D scenes, which is promising. Both qualitative and quantitative comparisons demonstrate its superior performance.
summary: This work studies the interactive segmentation task. Based on the prior knowledge that ground and background objects are located at different depths, this work proposes a new framework named OIS. OIS effectively takes advantage of a corresponding depth map via the proposed order- and object-aware attention. Experiments demonstrate that OIS effectively improves segmentation performance by fewer clicks and boosts inference speed.
weaknesses: Overall, I feel quite OK with this work, and there are no MAJOR weaknesses. See below for several suggestions and typos.

- It is better to visualize the five clicks in Fig. 1 for an improved presentation.
- Paragraph 2, Sec. 1, Page 1: Redundant '(' before 'RITM'.
- Paragraph 2, Sec. 1, Page 2: Better to delete 'However' before 'current methods fail to ...'.
- The blue dots in Fig. 3 are inconspicuous. Please consider another conspicuous color.

Meta-review: The paper receives 4 positive and 2 negative ratings after rebuttal, with 3 upgraded scores. Initially, the reviewers had several concerns about technical motivation/contribution, ablation study, handling multiple objects, robustness to depth maps, and experimental fairness. In the post-rebuttal discussion period, three reviewers were satisfactory with the authors' comments and raised the rating. After taking a close look at the paper, rebuttal, and discussions, the AC agrees with reviewers' feedback of the proposed method being novel and effective to perform interactive segmentation. Therefore, the AC recommends the acceptance rating.

Decision: Accept (Poster)