Interactive Segmentation by Considering First-Click Intentional Ambiguity

Authors: image  encoder
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 07:40:57

Abstract: Interactive segmentation task aims at taking into account the influence of user preferences on the basis of general semantic segmentation in order to obtain the specific target-of-interest. Given the fact that most of the related algorithms generate a single mask only, the robustness of which might be constrained due to the diversity of user intention in the early interaction stage, namely the vague selection of object part/whole object/adherent object, especially when there's only one click. To handle this, we propose a novel framework called Diversified Interactive Segmentation Network (DISNet) in which we revisit the peculiarity of first click: given an input image, DISNet outputs multiple candidate masks under the guidance of first click only, it then utilizes a Dual-attentional Mask Correction (DAMC) module consisting of two branches: a) Masked attention based on click propagation. b) Mixed attention within first click, subsequent clicks and image w.r.t. position and feature space. Moreover, we design a new sampling strategy to generate GT masks with rich semantic relations. The comparison between DISNet and mainstream algorithms demonstrates the efficacy of our methods, which further exemplifies the decisive role of first click in the realm of interactive segmentation.

Main paper: 
 related work 2.1 interactive segmentation: 
 Interactive segmentation (IS) takes account of the human guidance to provide single, class-agnostic instance mask, which has been a long-standing topic since the advent of Intelligent Scissor [36] in 1995. During that period, researchers mainly focus on energy optimization methods, e.g., GrabCut [1] and random walk [10] . Those methods leverage low-level feature only, which is sub-optimal when confronting complex scenes. In 2016, the first deep-learning based algorithm DIOS [45] makes a remarkable breakthrough in IS. In this work, positive and negative clicks are encoded into two-channel distance maps concatenated with input image, an arbitrary segmentation model (e.g., FCN [31] ) takes it as input to get final result in an end-to-end manner. Later, ITIS [33] proposes iterative training strategy to simulate real-word interaction process in training phase, which is then revisited and further modified by [41] . In terms of interaction type, DEXTR [34] uses extreme points in four directions to indicate a compact range of object compared to click, while in IOG [48] , a bounding box is drawn for coarse localization, then clicks are added inside the box to obtain fine-grained mask. In recent studies, Next, a mask is selected based on user preference or the largest IoU score with GT, which will be further refined in 𝑅 using whole clicks.attention-based design has gradually become mainstream. CDNet [4] disseminates pixel features located at positive/negative clicks to other pixels using self-attention. iFPN [47] adopts sparse GNNs to propagate click information in a long-range manner. SimpleClick [27] and iSegFormer [28] are the pioneering works to combine transformer with IS, which greatly motivate relevant research such as iCMFormer [20] and InterFormer [16] .Originated from LD [21] , researchers attempt to combine IS with multiple choice learning [12] , which is a strategy that guides the model to generate more than one feasible solutions and select from one of these. In LD, the number of output channels is altered to 6, meanwhile only the minimum loss among those channels is backpropagated so as to learn discrepancy within channels. MultiSeg [22] provides multiple scale-aware masks by computing loss in anchor-truncated areas. SAM [18] utilizes a specific prompt token as a signal to judge whether to return diverse or single mask based on the level of ambiguity (number of click), PiClick [46] enables 7 object queries to learn output diversity under the supervision of multiple GT masks. In comparison, our method generates proposals only when user clicks for the first time. Even though clicks up to 2~3 rounds may still contain intentional uncertainty, we argue that a trade-off between low-time cost and performance uprising is of great necessity in network design.IS methods concerning about the utility of first click are quite few till now. In FCANet [26] , first click is regarded as a coarse prior for subsequent clicks to refine, in which they supervise first-clickonly prediction mask as a subtask using auxilliary loss. EMC-Click [7] improves FCANet by proposing two novel correction modules which boosts performance. The uniqueness of first click is still lack of sufficient exploration and it is meant to bring about a brand-new perspective when diving into relevant research. 
methodology 3.1 preliminary and overview: 
 Given an input image I ∈ 𝑅 𝐻 ×𝑊 ×3 with user-annotated set of pixelsC = {(𝑢 𝑖 , 𝑣 𝑖 , 𝑝 𝑖 )|𝑖 = 1, 2, • • • , 𝑐}, where (𝑢 𝑖 , 𝑣 𝑖 ) ∈ [0,𝑊 ] × [0, 𝐻 ]and 𝑝 𝑖 ∈ {0, 1} denotes the coordinates and property (i.e., 𝑝 𝑖 = 1 for positive and 𝑝 𝑖 = 0 for negative) of the 𝑖 𝑡ℎ click, an encoding function 𝐸𝑛𝑐2𝐷 (•) converts those clicks into a 2D pattern, e.g., distance map (denoted as S ∈ 𝑅 𝐻 ×𝑊 ×2 ), which is concatenated with image to form a 5-channels input. In a standard segmentation network 𝑓 , a pretrained image encoder scales down the input resolution by using stride-2 convolutions or pooling (e.g., ResNet-50), followed by stacks of conv layers to extract high-level feature, which is then upsampled by a decoder structure for semantic comprehension. The final output is a sigmoidized mask M ∈ [0, 1] 𝐻 ×𝑊 which indicates the fine-grained location of user-interest object. Later, user is allowed to add more clicks targeted on the mislabeled region, causing the iterative change of prediction together with input clicks. Therefore, we formulate the classical interactive segmentation pipeline as follows:EQUATIONwhere 𝑡 ∈ {1, 2, • • • ,𝑇 } denotes the 𝑡 𝑡ℎ interaction round, 𝜃 𝑓 denotes the network parameter. However, Formula 1 is insufficient when conditioned on multiple predictions combined with the prominence of first click, which characterizes the main architecture of DISNet. As is shown in Figure 2 , the whole network is split into two stages for proposal generation and mask refinement, respectively. In the first stage (denoted as proposal network 𝑃), click in the first round (i.e., 𝑡 = 1) is distinguished from other clicks in C 𝑡 due to the diverse user intentions it brings about, denoted as C 𝐹 , which is then converted by 𝐸𝑛𝑐2𝐷 (•) to form a single-channel disk map S 𝐹 (Here we suppose that C 𝐹 is always positive so we omit the second channel for negative clicks, since user is fond of clicking around the center of object at first, according to [26] ). Similarly, we take the image I with S 𝐹 as input, deliver to the image encoder, while a Mask2Former decoder is used to generate 𝑁 diverse, ambiguity-aware mask proposals M ∈ [0, 1] 𝐻 ×𝑊 ×𝑁 using trainable object queries Q ∈ 𝑅 𝑁 ×𝑑 , together with a pixel decoder which produces multi-scale features on the basis of encoder output. Finally, a specific mask M 𝑠 is selected from M as input to stage 2 (denoted as refinement network 𝑅) to receive further correction. The above procedure can be formulated as follows:EQUATIONwhere 𝜙 𝑠 (•) denotes mask selection principle (see Section 3.2), note that Q is part of 𝜃 𝑝 as network parameters hence absence in the variable list. It is obvious that M 𝑠 remains solid after the first click is given, which won't be altered w.r.t. C 𝑡 or M 𝑡 -1 . Therefore, DISNet only needs to focus on mask refinement from the second round, which enormously reduces the cost of dense matrix operation in stage 1.Next, subsequent click is added in order to refine the erroneous part of M 𝑠 . The object query Q 𝑠 ∈ 𝑅 𝑑 (takes charge of predicting M 𝑠 ), together with 4x resolution feature map produced by pixel decoder (denoted as F -1 ∈ 𝑅 𝐻 /4×𝑊 /4×𝑑 ), are reused in stage 2 for consistency. Specifically, two elaborately designed attention module followed by respective type of FFNs act as a modified mask decoder in SAM to output the final mask M 𝑓 , which is formulated as below:EQUATIONwhere P 𝑡 ∈ 𝑅 𝐶 ×𝑑 is generated using 𝐸𝑛𝑐1𝐷 (•) function to get a linear representation of C 𝑡 . We'll clarify this function and the detailed mechanism of stage 2 in Section 3.3.Masks with all possible semantic combinations (user intentions) will be provided in this stage. We mainly follow the design of PiClick [46] except for some minor adjustments. Image encoder. A general encoder is often used to extract image feature as a preprocess in almost all computer vision task. In our method, a plain ViT pretrained with Masked Image Modeling (MIM) [14] is adopted, which includes a patch embedding layer and several window-based multi-head self-attention layers (W-MSA). We obtain feature with 16x resolution and 784 channels (ViT-B version) from the encoder. Mask decoder. We constitute our mask decoder with a pixel decoder to get multi-level features {F 𝑖 |𝑖 = 1, 2, 3}, plus several Mask2Former [5] decoder layers which involves: a) self-attention within object queries Q. b) cross-attention between Q and F 𝑖 . c) feedforward network (FFN). Predictions are obtained by matrix calculating (convolving) 4x feature F -1 with the updated Q, while a bipartite matching loss is measured between prediction masks and relation-aware GT masks (see Section 3.4). The specific mask to refine in stage 2 is picked out according to its largest IoU with all GT masks(during training and evaluation) or user selection (during inference). Note that we abandon the design of parallel IoU predictor (Target Reasoning Module) in PiClick [46] since it is an ill-posed problem to produce accurate IoU without external guidance under first-clickonly circumstance.Motivated by SAM [18] , we propose masked click attention module (regarded as modified image-to-token) with first-click guidance module (regarded as modified token-to-image) to progressively renovate image features and token, as illustrated in Figure 3 . Masked click attention. We start by introducing the masked attention scheme adopted in DETR series [5, 9] . Suppose we have 𝑄, 𝐾, 𝑉 that satisfies 𝑄 = 𝜑 (Q 𝑠 ), 𝐾 = 𝜓 (F -1 ), 𝑉 = Θ(F -1 ), where 𝜑, 𝜓 and Θ are linear transformations. In masked attention, the value of 𝑄𝐾 𝑇 is summed with a modulation term 𝑀 before softmax operation.EQUATIONThe above 𝑀 enforces all pixels in the background (sigmoid value less than 0.5) to be infinitesimal so that information flow is limited into foreground pixels only. When combined with our work, 𝑀 is replaced by log(M 𝑠 ), where M 𝑠 is the selected mask proposal based on first click. There're mainly two concerns: a) M 𝑠 serves as an initial template to provide coarse localization, which accelerates the convergence speed of training. b) M 𝑠 is solid, means the contextual continuity of first click could be kept. We use log(•) instead of binary threshold to ensure smooth transition, which could also contribute to efficient training. We also notice that positive/negative click is also the subset of foreground and background pixels. From the click perspective, mask is somewhat a zone of propagation or diffusive growth start from seeds (clicks). Therefore, another modulation term is required to manifest the saliency of clicks in masked attention. Hence, we generate two linear decay maps (denoted as G 𝑝 and G 𝑛 ) according to click positions, which satisfies:G 𝑝/𝑛 (𝑖, 𝑗) = 1 - min 𝑟 0 , 𝜙 C 𝑝/𝑛 , 𝑝 𝑖 𝑗 𝑟 0 (5)where C 𝑝/𝑛 is the positive/negative click set, 𝜙 (•) is euclidean distance, 𝑟 0 is a radius to control the rate of decay (We set 5 for other clicks and 15 for the first click). Lastly, the masked click attention (MCA) is formulated as below:EQUATIONEQUATIONEQUATIONwhere means Hadamard product. It is evident that G 𝑝/𝑛 is utilized to reweight 𝑄𝐾 𝑇 under the constraint of M 𝑠 , thus the feature of pixels nearby those clicks are the most likely to be gathered, which realizes delicate, oriented information flow from image to token. First-click guidance. We strive to forge a novel token-to-image module that satisfies: a) low computation cost. b) a thorough exploitation of the first click. Reviving Formula 3, clicks are encoded into 1D vectors P 𝑡 by initializing two trainable indicators representing the positive/negative property, a positional embedding function is used to transform clicks coordinates into a dense vector form, which is then added with indicator. Followed by concatenation and linear projection, one could obtain click-augmented features which we denote as follows:EQUATIONA latent issue is that clicks may be sub-optimal such as clicking in regions with blurred patches or low lighting, even the click itself could be erroneous due to user's negligence, i.e., mark negative on foreground. A non-trivial solution is to exclude those clicks or weaken their influence to the network. Naturally, the first click contains the user's most primary impression of object's global structure, which is qualified for guidance prior. Therefore, we measure the feature-space correlation between the first click and other clicks (denoted as relational vector V) in order to pose a constraint that only the click which shares similarity with the first click could contribute to mask output. To this end, we split F ′ into F ′ 𝑓 𝑐 , F ′ 𝑝𝑜𝑠 and F ′ 𝑛𝑒𝑔 , then cross-attention within the three is computed as follows, where MHA means multi-head attention: (10) Later, we integrate V with image feature F -1 by utilizing SENet [15] , which consists of two fully-connected layers, a sigmoid function, and a channel-wise multiplication between V and F -1 . This is fairly efficient compared with the original token-to-image module in SAM, in which a spatially cross-attention between F -1 and all clicks embedding is implemented. Similar to MCA, we use M 𝑠 as a second constraint to limit the scope of V. Thus, the first-click guidance (FCG) module is formulated as below:V 𝑝𝑜𝑠/𝑛𝑒𝑔 = MHA F ′ 𝑓 𝑐 , F ′ 𝑝𝑜𝑠/𝑛𝑒𝑔EQUATION)EQUATION)EQUATIONFinally, we adopt a depth-wise ConvFFN module to make up for the lack of spatial attention, which consists of two 1 × 1 conv layer and a 3 × 3 depth-wise separable conv layer. This is a common practice in many real-time vision transformers.The generation of diverse ground-truth masks is a non-trivial task for the supervision of proposal network in DISNet. Based on SBD [13] and LVIS [11] datasets, we attempt to discover the best strategy to measure the possible spatial/semantic correlation within all masks in an image, which we categorize into proximity-based method and hierarchy-based method. A concrete algorithm description of mask sampling with clicks using the two methods is shown in supplementary material. Hierarchy-based method. Thanks to the hierarchy tree provided in LVIS that enable us to construct a sequence of object masks with rich semantic relations. In this scenario, a scene could be decoupled into different levels of scene nodes (may contain multiple objects for the root, and object part for the leaf) to insinuate the possible subject-predicate-object (SPO) relations among them. Concretely speaking, we adopt a bottom-up strategy where we first randomly pick out a center object (e.g., an apple) and locate its node level. Then we traverse all of its parent nodes (e.g., a man with an apple) using depth ordering, until the level of node reaches the top (e.g., a man with an apple is sitting on a chair). Finally, the center object along with parents are added into an empty list of diverse groundtruth masks in an inner-outer manner. We sample the first click based on the subtracted region between the center object mask and all of its children masks, while a random mask is selected to supervise the refinement network. Proximity-based method. Mask sampling using hierarchy tree is not feasible in SBD. Therefore, we simply judge whether two arbitrary masks are spatially proximate by calculating the overlapped pixels after a 3 × 3 mask dilation, which is not guaranteed for semantic relations. We use all combinations of 1-hop neighbors to get the list of diverse samples. 
experiments 4.1 datasets and evaluation metrics: 
 We conduct our experiments on the following datasets: GrabCut [39] , Berkeley [35] , DAVIS [37] , SBD [13] , PascalVOC [8] and a combination of COCO [24] and LVIS [11] . The GrabCut dataset contains 50 images with single object. The Berkeley dataset consists Table 1 : A comprehensive comparison between the mainstream algorithms and our method. Methods marked with ↑ means the first-click is treated specially during segmentation, while ↓ means methods with multiple outputs. The best result is marked with blue (trained with SBD) and red (trained with COCO+LVIS). It is obvious that our method possesses both of the two characteristics. [7, 27, 41] to randomly sample 345 frames for testing. The PascalVOC dataset contains 1449 testing images with 3427 instances. The COCO+LVIS is a compound dataset which consists of 118K images with high-quality annotations of about 1.2M instances. In our work, the SBD and COCO+LVIS are used for training purposes while the rest is for testing. We follow the commonly used evaluation protocol for interactive segmentation, including a) Mean intersection-over-union (mIoU), which measures the percentage of pixels in the overlapped regions of predicted mask and ground truth. b) Number of clicks (NoC), which measures the least number of clicks required to reach a given IoU threshold 𝑥 (denoted as NoC%𝑥, where 𝑥 takes the value of 85 or 90 as a common practice in previous work [4, 20, 29, 48] ).Following [46] , we adopt a ViT-B as backbone (pretrained by MAE) together with 3 Mask2Former decoder layers for proposal generation. The selection of proposal is based on the largest IoU with current ground-truth mask, or manually by user preference when the model is online for real world use. In the latter part of DISNet, the number of DAMC module is commonly set to 2 to prevent latent computation cost. During the training process, We supervise the whole network with a bipartite matching loss [2] for stage 1 and a normalized focal loss (NFL) [23] for stage 2, while the learning rate 𝑙𝑟 is set to 5𝑒 -5 and 5𝑒 -6 for two stages, respectively. We adopt Adam [17] optimizer with a momentum of 𝛽 1 = 0.9 and 𝛽 2 = 0.99, followed by a cosine-annealing scheduler [32] for progressive 𝑙𝑟 decay (we set the warmup step to 2 epochs, and initial 𝑙𝑟 to 5𝑒 -8 ). We train 55 epochs on SBD dataset and 70 epochs on COCO+LVIS dataset, both using a batch size of 32. For a training sample, we randomly crop and resize it to a resolution of 448 × 448, followed by classical data augmentations, i.e., random brightness/contrast, horizontal/vertical flipping, etc. All the experiments are conducted on Ubuntu-18.04 platform with 4 RTX 4090 GPUs, while our main code is constructed on two open-source projects (i.e., MMSegmentation [6] and RITM [41] ).During inference when the first click is positioned, we truncate the path to refinement network 𝑅 so that the selected mask (output from proposal network 𝑃) is chosen for metric evaluation, since we notice a subtle performance drop around 0.3 NoC if we evaluate the refined mask from network 𝑅 instead. Clearly, only the first click itself provides limited prior knowledge to fix local details. Starting from second click, we reuse the output from network 𝑃 such that only 𝑅 is utilized, which avoids redundant computation cost. This time-efficient strategy is similar to [7] .We make a thorough comparison with the mainstream IS algorithms, the Number of Click (NoC) result is shown in Table 1 . To be fair, we don't emphasize on the effect of a stronger backbone or a strong dataset so we group methods with the above similar setting for convenient analysis. We notice that our work outperforms most of the previous works, where there's a slight improvement of 0.02 to 0.08 compared with PiClick on GrabCut and a huge improvement over EMC-Click [7] (about 0.2 to 0.3)-the latter method also attempts to measure the peculiarity of first click but failed to sufficiently leverage this information. When in terms of dataset, we argue that a high quality masks annotation is indispensable since we surpass our own counterpart (trained by SBD) at an astonishing value of 0.4 to 0.5. Reviving the hierarchy mask sampling strategy in COCO+LVIS dataset, it is likely to make great contribution not only for the proposal stage, but also a guidance prior to the final output. In terms of multiple output mask design, CDNet [4] utilizes an auxiliary branch to learn the correlated semantic, yet compared with PiClick [46] or our method, it fails to converge well enough due to the lack of diverse ground truth masks.Curves measuring the change of mIoU with respect to number of clicks are shown in Figure 4 . We make a brief comparison among up to 7 methods on GrabCut, Berkeley, SBD and DAVIS datasets, respectively. It is evident that our method performs the best especially at first click.We visualize the outputs from different stage in our framework, as is illustrated in Figure 5 . A specific proposal is selected based on largest IoU with ground truth, then further refined by successive clicks. We notice that it is capable of harvesting result with sufficient accuracy starting from the second click.We also strive to analyze two main properties in our work, i.e. first-click and ambiguity-aware mask proposals. As shown in 6, we visualize and compare the proposals (PiClick) or final result (FCANet, EMC-Click) at first click, together with IoU with respect to GT (red region in the middle). Our method tends to produce more accurate proposals compared to others (i.e., we segment the click position into sofa cushion/sofa/adjacent sofas). In this chapter, we conduct several ablation studies using Berkeley and DAVIS dataset, in which we take a deep view over each component of DISNet. Impact of whole network. We start by progressively adding core components to a plain baseline (Here we choose SAM [18] ), including FC (i.e., first-click) and MO (i.e., multiple output) design in proposal network 𝑃, together with MCA and FCG module in refinement network 𝑅. In addition to NoC, we utilize a novel metric called Number of Failure (NoF%𝑥) which measures the number of cases where the segmentation mask could not reach a given mIoU 𝑥% within the maximally allowed number of clicks (default is 20). Reviving result in Table 2 , we could notice that model with FC achieves a major breakthrough in NoC, which indicates the importance of first click. Furthermore, a proper integration of these four components leads to the best result, which means they're tightly organized with less redundancy during the whole pipeline. Impact of MCA. Targeted on the rate of decay around each click, we design a scheme to record the effect of MCA under different combination of radius value 𝑟 0 for first/other clicks (shown in Table 3 ). We conclude that setting this two value to 15 and 5 could yield the best result. Impact of FCG. To measure the utility of FCG. We split it into two components: a) whether to use click-augmented feature (simplified as AC) instead of image feature. b) whether to use ConvFFN module in its following. The result in Table 5 demonstrates that each component is indispensable to the final contribution of evaluation metrics. Impact of mask sampling. We apply SBD and COCO+LVIS training dataset with two relation-aware sampling methods, while we record our result in Table 4 . Evidently speaking, hierarchy-based method is more likely to produce accurate, semantic-exclusive diverse predictions than proximity-based method, which has been also demonstrated in Table 6 .  
conclusion: 
 In this paper, we propose a two-stage interactive segmentation method DISNet, where the peculiarity of first interaction click is highlighted as maximum intentional ambiguity, together with contextual continuity. A novel refinement network DAMC further corrects details of the selected mask from the proposal network, which proves robust and efficient as well. Moreover, our proposed diverse ground-truth sampling strategy plays a crucial role in realworld simulation of user intentions. Extensive comparison and ablation studies demonstrates state-of-the-art performance on several datasets, which further exemplifies the decisive role of first click in the realm of interactive segmentation. Interactive Segmentation by Considering First-Click Intentional Ambiguity 
: 
 infrared image) and interaction modalities (e.g., click [7, 21, 43, 46] , scribble [3] , extreme points [34, 48] , contour [1, 19, 36, 38] , prompt [18, 44, 49] ), which have also been popularized in data annotation, autonomous driving, as well as medical image analysis.Over the last few years, researchers have made great efforts to enhance the classical interactive segmentation pipeline using prevailing computer-vision techniques. A large proportion of latest methods are embedded with attention mechanism (e.g., transformer), which proves conducive for long-range context modeling. The pioneering work SimpleClick [27] adopts an MAE-pretrained ViT [14] consisting of 12~32 W-MSA layers [30] ; SAM [18] and SEEM [49] also use ViT to pre-compute image features, while they further measure the cross-attention between image feature and encoded prompts in a two-way manner, thus facilitates mutual information flow.It is also noteworthy that user ambiguity is another crucial aspect which has been gradually brought into focus. The meaning of ambiguity is that a single interaction may correspond to multiple system feedback, given that the annotated pixels by users might contain diversified semantics, which is ubiquitous in object/set of objects that is commonly characterized by structural hierarchy or spatial adjacency, such as desk vs. book on the desk or man vs. camera in man's hand. In terms of interaction modality, the mainstream click-based methods become susceptible to vague selection of mask prototypes compared with scribble-based or contour-based counterparts, since click possesses the sparsest prior knowledge that it is hard to indicate the accurate range of a specific object. Previous methods such as LD [21] and MultiSeg [22] attempts to tackle this issue using additional output channels, while recent study PiClick [46] reformulates this task on the basis of DETR [2] , using solid number of object queries to generate mask proposals, which greatly surpasses the former. However, its weakness is also obvious: a) The rapid decline of ambiguity in the early rounds leads to output convergence (diverse to single), which causes the waste of object queries and its highly time-consuming computation (mostly 𝑂 (𝑛 2 )). b) The generation of diverse GT masks is based on random merging strategy, hence quite a few semantic or spatial correlation between those masks, which hinders performance.Our work makes further investigation into how to elaborately design an architecture where diversified user intentions are sufficiently parsed during interactive segmentation. We revisit and summarize two peculiar features of first click: a) Maximum ambiguity, means the number of diverse mask predictions reaches a peak when there's only one click. b) Contextual continuity, means the role of subsequent clicks is to refine details based on selected mask proposal (serves as initial template). We propose Diversified Interactive Segmentation Network (DISNet), a novel framework that decouples the classical segmentation procedure into two sections-proposal network and refinement network (see Figure 1 ). The main body of proposal network is Mask2Former [5] , in which All-clicks refinement we use RGB image combined with 1-channel first click map as input to get diverse mask predictions. Later, a single mask is picked out according to the largest IoU between predictions and GT mask, then the selected mask/mask token, image feature, together with encoded full clicks are sent to refinement network, where we introduce Dual-attentional Mask Correction (DAMC) which can be regarded as a variant of two-way transformer in SAM, including a masked click-attention module and a first-click guidance module. The former updates mask token using cross-attention between token and image feature, where the affinity matrix is reweighted by endowing its elements in the proximity of clicks with a larger value to emphasize click propagation. The latter updates image feature by computing a relational vector between first click and other clicks, then utilizes it to perform channel-wise activation with image feature. Both modules aim to strengthen cross-over information flow in terms of robustness and efficiency. Finally, we design a novel principle to generate sequence of diverse groundtruth masks on SBD [13] and LVIS [11] datasets. Evaluation on six benchmarks shows outstanding performance compared with the existing methods. We achieve 3.07 NoC%85 with 5.11 NoC%90 when trained with LVIS, which outperforms PiClick [46] (3.11 NoC%85 with 5.32 NoC%90), which is the current SOTA method. We summarize our contributions as follows:• We introduce an interactive segmentation framework DIS-NET that features multiple-output and first-click design, in which we fully exploit the properties of user's first click: a) to represent latent, diverse user intentions (maximum ambiguity). b) to guide and constraint the impact of successive clicks (contextual continuity). • We propose Dual-attentional Mask Correction (DAMC) component, a modified two-way transformer used to mutually measure the attention among image features, click features and selected token/mask proposal, which is proved to be capable of manifesting the decisive effect of user's first click in terms of information flow. • We propose a novel mask-sampling method in order to match our new framework. During training, we generate a set of semantic-correlated ground-truth mask proposals given the position of first click, which are then used to supervise the output from the first stage of DISNET. • Extensive comparisons with former works, visualization and ablation studies have demonstrated the necessity of our network design from macro to micro level. We conduct these experiments on six datasets using two evaluation metrics. 
 
 Appendix:  


Decision: The paper presents a promising approach to interactive image segmentation that demonstrates good performance in extensive evaluations. The most critical issue, highlighted by all reviewers, is the clarity of the evaluation methodology and the limited novelty. Reviewer uD9Y (and the others generally) pointed out a significant flaw in how the automatic selection of masks is handled, which could potentially undermine the credibility of the results. The authors' rebuttal included a detailed explanation of how this selection process is accounted for in the interaction metrics, ensuring that the evaluation accurately reflects real-world usage scenarios. This is very critical content, that must be updated in the manuscript. Nonetheless, reviewers praised the results, that the proposed framework improves the results already available in literature, with few exceptions on specific cases, where the proposed framework obtains state-of-the-art performance anyway. Considering this, and the general suggestion of acceptance from the reviewers, I recommend the paper for Acceptance. I strongly request that the authors update the paper to ensure the critical issue is corrected in the camera ready version of the paper.

Reviews: review: paper_summary: paper_summary: The authors present their "DISNet" (Diversified Interactive Segmentation Network), a framework for interactive segmentation that 1) tries to enhance the exploitation of the first click and 2) maximize the impact of subsequent clicks. In addition to this key contribution, the authors provide a sound experimental evaluation of their segmentation framework with six different datasets as well as a comparison with many different alternatives available in literature. They outperform most of them and their framework is very near to the few methods that do better of it in one of the datasets, in terms of number of clicks required to reach an intersection over union of 85% or 90%. The authors provide an ablation study as well.
strength_weakness: strength_weakness: 1. The empirical evaluation is pretty sound: 2 different datasets for training, 4 for testing, comparison with many different algorithms available in literature.

2. The results show that the proposed framework outperforms/improves the results already available in literature, with few exceptions on specific cases, where the proposed framework obtains state-of-the-art performance anyway.

3. Sound description of the methodology
limitations: limitations: 1. The main weakness in this work is the novelty. In fact, approaches that aim to reduce the clicks are available in the literature, even based on two stage architectures (in a similar fashion to what is proposed in this paper). For example:
 - T. Wang, H. Li, Y. Zheng and Q. Sun, "One-Click-Based Perception for Interactive Image Segmentation," in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2023.3274127
 - J. Lin et al., "AdaptiveClick: Click-Aware Transformer With Adaptive Focal Loss for Interactive Image Segmentation," in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2024.3378295
 - K. Li et al., "Multi-granularity Interaction Simulation for Unsupervised Interactive Segmentation," 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023, pp. 666-676, doi: 10.1109/ICCV51070.2023.00068

Therefore, the authors should discuss these works and highlight the difference with their approach and compare against them as well.

2. I think that the paper is missing a discussion about the computation required by the authors' architecture with respect to those they compare against. At least, it might be in the form of a comment/speculation given that computing execution time might be unreliable given that the hardware used by the authors' might be different from the hardware used in other works..

Minor: in the ablation study the authors mention table 6 that is not in the paper.
paper_summary: paper_summary: The manuscript presents DISNet, a pioneering interactive image segmentation framework that capitalizes on the inherent ambiguity of the initial user click to craft an array of initial mask proposals. These proposals are meticulously honed through a Dual-attentional Mask Correction module, complemented by an innovative sampling strategy to produce a spectrum of diverse ground truth masks. The framework's enhanced performance underscores the pivotal role of embracing the nuances of the first click. Nonetheless, the paper has certain ambiguities that necessitate a thoughtful and precise response from the authors.
strength_weakness: strength_weakness: 1. It is valuable to recognize the ambiguity of the first click, and the analysis conducted in the paper is commendable.
2. The majority of the paper is clearly written and easy to understand.
limitations: limitations: 1. The paper presents a significant flaw in its evaluation methodology: The automatic selection of the mask with the IoU from the proposals should be counted as an interaction because, in actual inference, it would require user operation and cannot rely on IoU computation. However, the authors have not clearly stated whether this selection is counted as an interaction in their reported performance metrics. This lack of clarity undermines the credibility of the results.
﻿
2. The first click is typically placed at the center of an object, which inherently minimizes part ambiguity. The advantage of this approach seems to be most effective when there are multiple candidate regions with centers that are in close proximity. Therefore, a more detailed comparative analysis with FCANet is warranted. The performance should be specifically examined in scenarios where the centers of the candidate regions are both close and distant to each other.
paper_summary: paper_summary: To solve the ambiguity caused by the first click in the task of interactive image segmentation, the paper propose a novel framework called DISNet to better handling the peculiarity of  the first click.
strength_weakness: strength_weakness: 1.The performance seems good.
2.This is the first work to treat the first click specially and output multiple result.
limitations: limitations: The biggest issue is that the results obtained after the first click in this paper were selected using the IoU metric, which assumes that the user has already obtained GT, which is not consistent with real-world scenarios. 
Therefore, the performance data in this paper are unreliable. For example, the NoC of GrabCut is 1.16, It is based on the user selecting the best initial mask, which means that there is at least an additional interaction amount exceeding one point in the selection process. However, this interaction amount is not calculated in the SOTA table, so the SOTA data in the table is extremely unreasonable.
If the author can provide a reasonable explanation for this, I can consider changing my score.

Format problems:
1.Maintain adequate line length by filling each line with at least one-third of its content.
2.It's recommended to prioritize vector graphics over rasterized images for figures unless unavoidable.
3.unify the "first-click" and "first-click"