AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation
Authors: Yuanwen Yue
Sabarinath Mahadevan
Jonas Schult
Francis Engelmann
Bastian Leibe
Konrad Schindler
Theodora Kontogianni
Venue: ICLR 2024 poster (2024)
Keywords: interactive segmentation
3D instance segmentation
point cloud
attention
Primary area: applications to robotics, autonomy, planning

Abstract: During interactive segmentation, a model and a user work together to delineate objects of interest in a 3D point cloud. In an iterative process, the model assigns each data point to an object (or the background), while the user corrects errors in the resulting segmentation and feeds them back into the model. The current best practice formulates the problem as binary classification and segments objects one at a time. The model expects the user to provide positive clicks to indicate regions wrongly assigned to the background and negative clicks on regions wrongly assigned to the object. Sequentially visiting objects is wasteful since it disregards synergies between objects: a positive click for a given object can, by definition, serve as a negative click for nearby objects. Moreover, a direct competition between adjacent objects can speed up the identification of their common boundary. We introduce AGILE3D, an efficient, attention-based model that (1) supports simultaneous segmentation of multiple 3D objects, (2) yields more accurate segmentation masks with fewer user clicks, and (3) offers faster inference. Our core idea is to encode user clicks as spatial-temporal queries and enable explicit interactions between click queries as well as between them and the 3D scene through a click attention module. Every time new clicks are added, we only need to run a lightweight decoder that produces updated segmentation masks. In experiments with four different 3D point cloud datasets, AGILE3D sets a new state-of-the-art. Moreover, we also verify its practicality in real-world setups with real user studies. Project page: https://ywyue.github.io/AGILE3D.

Published as a conference paper at ICLR 2024
AGILE3D: ATTENTION GUIDED
INTERACTIVE
MULTI-OBJECT 3D SEGMENTATION
Yuanwen Yue 12
Sabarinath Mahadevan Jonas Schult
Francis Engelmann" 2.4
Bastian Leibe
Konrad Schindler Theodora Kontogianni
PPhtoogrammetry and Remote Sensing, ETH Zurich CComuter Vision Group, RWTH Aachen University
2TTH Al Center, ETH Zurich Goooge
ABSTRACT
During interactive segmentation, a model and a user work together to delineate objects of interest in a 3D point cloud. In iterative process, the model assigns each data point to an object (or the background), while the user corrects errors the resulting segmentation and feeds them back into the model. The current best practice formulates the problem as binary classification and segments objects one at time. The model expects the user to provide positive clicks indicate regions wrongly assigned to the background and negative clicks on regions wrongly as- signed the object. Sequentially visiting objects is wasteful since it disregards synergies between objects: a positive click for given object can, by definition, serve as negative click for nearby objects. Moreover, a direct competition be- tween adjacent objects can speed up the identification of their common bound- ary. We introduce AGILE3D, an efficient, attention-based model that (1) supports simultaneous segmentation of multiple 3D objects, (2) yields more accurate seg- mentation masks with fewer user clicks, and (3) offers faster inference. Our core idea is encode user clicks spatial-temporal queries and enable explicit inter- actions between click queries as well as between them and the 3D scene through click attention module. Every time new clicks are added, only need to a lightweight decoder that produces updated segmentation masks. In experiments with four different 3D point cloud datasets, AGILE3D sets new otate-of-the-art. Moreover, we also verify its practicality in real-world setups with real user studies. Project page: https://ywyue http://wwww..ggttbbo/ooooo github.io/AGILE..
INTRODUCTION
Accurate 3D instance segmentation is crucial task for a variety of applications in computer vision and robotics. Manually annotating ground tround-truth segmentation masks on 3D scenes is expensive and fully-automated 3D instance segmentation approaches (Hou et al., 2019; Engelmann et al., 2020; Chen et al., 2021; Vu et al., 2022) do not generalize well to unseen object categories of an open- world setting. Interactive segmentation techniques have been commonly adopted for large-scale 2D image segmentation (Benenson et al., 2019), where a user interacts with a segmentation model by providing assistive clicks or scribbles iteratively. While interactive image segmentation has been the subject of extensive research (Xu et al., 2016; Li et al., 2018; Mahadevan et al., 2018; Jang & Kim, 2019; Kontogianni al., 2020; Sofiuuk et al., 2022; Chen al., 2022), there are only few works that explore interactive 3D segmentation (Valentin et al., 2015; Shen et al., 2020; Zhi al., 2022). Those methods either only work for semantic segmentation or require images with associated camera poses w.r.t. the 3D scene. The latter constraint makes them not suitable for non-camera sensors (e.g., LiDAR), or for dynamically changing viewpoints like in AR/VR settings. Even with available images, providing feedback for the same object from multiple viewpoints can be tedious. Here, we focus on interactive segmentation directly in 3D point clouds.
Recently Kontogianni et al. (2023) introduced approach for interactive 3D segmentation that operates directly on point clouds, achieving state-of-the-art performance. The task is formulated as binary segmentation for one object at a time (Fig. 1, left), and follows the mainstream approach of interactive 2D image segmentation (Xu et al., 2016; Mahadevan et al., 2018; 8; Li et al., 2018; Jang & Kim, 2019; Kontogianni et al., 2020; Sofiiuk al., 2022; Chen et al., 2022): User input comes in the form of binary clicks that identify locations with incorrect labels. Positive clicks specify missed parts of the target object and negative clicks specify background areas falsely labeled as

---

Published as a conference paper at ICLR 2024
de O0ee00
Newr Cl&ck
Next
Scona Per Click
Nevwrk
XVZ (8GB))
odfs dloee
Tredindd Hark D'ger St
((GB)
0.55
Prokksd Mack
Taltum
Veni
Figure 1: Architecture comparison. Left: InterObject3DD (Kontogianni et al., 2023). Right: our AGILE3D. Given the same set of 10 user clicks, AGILEED can effectively segment three objects while InteObject3D can only segment one. InterObject3D takes 0.5s for one object while only 0.25s for all three, thanks to running a lightweight decoder per iteration and not a full forward pass through the entire network'. / Unlike InterObject3D, our backbone learns features for all objects.
need
foreground. The two sets of click coordinates are encoded as binary masks and concatenated with the 3D point coordinates to form the input to deep learning model. A subtle, but important limitation of that approach is that objects are processed sequentially, one at a time: obviously. object outlines within scene are boundaries between different objects (rather than between isolated object and a monolithic background): Positive clicks for one object can, by definition, serve as negative clicks for other, nearby objects. In most cases also the opposite is true, negative clicks will often also lie on other objects of interest. Handling multiple objects jointly rather than sequentially also brings a computational advantage, because every round of user input requires forward pass.
In this work, we propose AGILE3D, an attention-guider guidedd interactive 3D segmentation approach that can simultaneously segment multiple objects in context. Given 3D scene, we first employ stan- dard 3D sparse convolutional backbone to extract per-point features without click input. Instead of encoding clicks as click maps, we propose to encode them as high-dimensional feature queries through our click-as-query module. To exploit not only the spatial locations of user clicks but also their temporal order in the iterative annotation process, they are supplemented by a spatial-tempora positional encoding. To enable the information exchange between different clicks, and between clicks and the 3D scene, we propose a click attention module, where clicks explicitly interact with the 3D scene through click-to-scene and acene-to-click attention and with each other through click- to-click attention. Finally, we aggregate the roles of all click queries to obtain a single, holistic segmentation mask in our query fusion module, and train the network in way that regions compete for space. In this manner, AGILE3D imposes no constraint on the number of objects and seamlessly models clicks on multiple objects, including their contextual relations allowing for more accurate segmentation masks of multiple objects together. Disentangling the encoding of the 3D scene from the processing of the clicks makes it possible to pre-compute the backbone features, such that dur- ing iterative user feedback one must only run the lightweight decoder (i.e. click attention and query fusion), thus significantly reducing the computation time (Fig. 1). Moreover, our backbone learns the representation of all scene objects while InterObject3D only learns the representation of a single object depending on the input clicks (c.f. learned features converted to RGB with PCA in Fig. 1).
Our proposed method aims to be trainable with limited data, and able to compute valid segmentation masks for zero-shot and few-shot setups on unseen datasets (e.g., dataset annotation). We train on single dataset, ScanNetV2-TT -rrrinn (Dai et al., 2017), and then evaluate on ScanNetV2-VV Val (Inc. SeanNet20 and ScanNet40) (Dai et al., 2017), S3DIS (Armeni et al., 2016), KITTI-360 (Liao et al., 2022). For all of them, AGILE3D outperforms the state of the art. Our main contributions are:
1. We introduce the interactive multi-object 3D segmentation task to segment multiple objects concurrently with limited number of user clicks in 3D scene.
2. We propose AGILE3D, which is the first interactive approach that can segment multiple objects in 3D scene, achieving state-of-the-art in both interactive multi- and single-object segmentation benchmarks.
3. We propose the setup, evaluation, and iterative training strategy for interactive mujti- segmentation on 3D scenes and conduct extensive experiments to validate the benefits of our task formulation.
4. We develop a user interface and perform real user studies to verify the effectiveness of our model and the proposed training strategy in real annotation tasks.
'Time measured single TITAN RTX GPU.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
3D instance segmentation. Fully-supervised 3D instance segmentation is well-researched prob- lem with remarkable progress (Hou et al., 2019; Yi et al., 2019; Engelmann et al., 2020; Jiang et al., 2020; Vu et al., 2022; Schult al., 2023; Sun al., 2023). Like all fully-supervised methods, they require large amounts of annotated training data. To relieve the labeling cost, Hou et al. (2021); Chibane et al. (2022) explore weakly-supervise annotations such as sparse points or bounding boxes with the cost of performance drop. For survey on 3D segmentation, we refer the readers Guo et al. (2020); He et al. (2021); Xiang et al. (2023). Interactive 3D segmentation differs from those methods. First, fully and weakly weakky--upervvsed methods require per dataset training and are unable to generalize to classes that are not part of the training set. Moreover, they cannot incorporate additional user input to refine any inaccuracies further, whereas our method aims to produce high-quality masks using the model's ability to interact with humans.
3D instance segmentation by learning from weak
Interactive 3D segmentation. There are only a few approaches that support user input in gener- ating 3D segmentation masks (Valentin et al., 2015; Shen et al., 2020; Zhi et al., 2022; Kontogianni et al., 2023). Valentin et al. (2015); Zhi et al. (2022) focus on online semantic labeling of 3D scenes rather than instance segmentation. Shen et al. (2020) shift the user interaction the 2D domain but require images with known camera poses. Moreover, providing feedback for the same object in multiple viewpoints is cumbersome. The closest work ours is InterObject3D (Kontogianni al., 2023), an interactive 3D segmentation approach that directly operates on point clouds. However, In- terObject3D can only segment single objects sequentially whereas our method can segment multiple objects simultaneously, leading to better results with less user interaction.
Interactive image segmentation has been extensively studied (Xu et al., 2016; Li et al., 2018; Mahadevan et al., 2018; Jang Kim, 2019; Benenson et al., 2019; Kontogianni
al., 2020; Chen et al., 2022; Sofiiuk et al., 2022; Liu et al., 2022; Du et al., 2023; Zhou et al., 2023; Wei et al., 2023) but all of the methods are designed to segment single objects. Only few works (Agustsson et al., 2019; Rana al., 2023) explore interactive full-image multi-object segmentation but specialize in the 2D domain. To the best of our knowledge, our method is the first approach that supports interactive multi-object segmentation in 3D point clouds. Moreover, although simulated evaluation is a well-established protocol in both the 2D and 3D domains, we urge the community to move beyond simulated clicks and also evaluate with real user studies.
METHOD
Consider a 3D scene P E NxC, with N number of 3D points and C the feature dimension sociated with each point. C' is normally set to 3 for locations zyz, otherwise 6 if colors rgb are available.
Interactive single-object segmentation. Given such a scene, in interactive single-object segmen- tation (Kontogianni et al., 2023), the user provides sequence of clicks, where positive clicks are considered on the desired object and negative clicks on the background. The segmentation mask is obtained through an iterative process: the model provides the user with segmentation mask, then the user provides feedback to the model via positive/negative clicks. The model provides an updated mask given the user corrections. The process repeats until the user is satisfied with the result.
Interactive multi-object segmentation. We extend the above formulation to incorporate interac- tive multi-object scenarios. Let us assume user wants to segment M target objects in scene. We denote user clicks as S = (2,,Co..... <=) where k is the click index. k: also doubles as a timestamp indicator since the clicks come as sequence in time. Each click C₁ is represented by two attributes C = {Pk: 0x where Pę E R3 are the 3D coordinates and og E {0, 1,., is the region index, indicating whether the click comes from the background (when oş = or associated with object og (when 0g > 1). S is initialized as an empty sequence and iteratively extended when the user gives more clicks. Given the 3D scene P and click sequence S, our goal is predict single, holistic segmentation mask M E 1, M)N which indicates the interest region each point belongs to. Please note we aim to ensure that each point belongs to single segmentation mask which is different from the interactive single-ohject segmentation task, where several passes of the same scene with different objects of interest might result in some points assigned to multiple segmentation masks. When M = 1, the above formulation matches the interactive single-object segmentation setting as in Kontogianni et al. (2023).
3

---

Published as a conference paper at ICLR 2024
Feature Back
Initial
Refined
Feature::
Features F2
Click
Bals------e Cleck Amt
Mask Hiad
Softwer
Cldk-a-uun Query
Mak M
Inttial Cluck Queriess
Refined
Click
Click Mask
Mask
Clicks: (b) Click as- Module
Click Attention Module
Quary Fusion Module
Figure 2: Model of AGILE3D. Given a 3D scene and user click sequence. (a) the feature backbone extracts per- per-point features and (b) the click-as-query module converts user clicks to nigh-dimensional query vectors. (c) The click attention module refines the click queries and point features through multiple attention mecha- nisms. (d) The query fusion module first fuses the per-elick mask logits to region-specific mask logits and then produces a final mask through a softmax. With - we denote the user click information and with + the scene information. Colors of clicks, click queries and segmentation masks are consistent for the same object.
3.1 AGILE3D
Our overall architecture (Fig. 2) consists of (a) a feature backbone that extracts per-point features, (b) click-as-query module that converts user clicks to apatial-temporal query vectors, (c) click at- tention module that enables explicit interaction between click queries themselves and scene features, and (d) query fusion module that predicts holistic segmentation of multiple objects.
Feature backbone. Our feature backbone is a sparse convolutional U-Net, based on the Minkowski Engine (Choy et al., 2019) as in Kontogianni et al. (2023). It takes as input a 3D scene and produces a feature map F E R=* xD where D is the feature dimension. Unlike Kontogianni et al. (2023) which sends the concatenation of P and encoded click maps to the backbone, we only input P and insert the user clicks separately in our bock-as-query module.
Click-as-query module converts each user click O to query qi- A click is user query indicating desired region. The click query should properly encode representative knowledge of object so that the system can correctly classify all relevant points. On the other hand, clicks are sequence of 3D points that are inherently spatial and temporal. Motivated by this, we encode the click query as three parts Qs= = {ct. Sk. } each of which models its content, spatial, and temporal properties. The content part C₁ E IRP is initialized from the point feature map F of the backbone. For each click, we find the nearest voxel position in F and use the feature of that voxel as Cg- The spatial part Sy E ERDi is created by mapping the click coordinates positional encodings (Tancik et al., 2020). Similarly, we transform the timestamp & to a temporal embedding t2 E RDD using a sin/cos positional encoding of (Vaswani et al., 2017). We consolidate the spatial and temporal parts positional part by summing Sę and tą- After initialization, the content part of each query along with per-point features F will be iteratively refined through the click attention module by interacting with each other.
to the same feature space as C₁ using Fourier
Click atfention module is designed enable interaction between the click queries themselves and between them and the point features. Each decoder layer consists of click-to tocc-tt--ceene attention module (C2S), a click- to-click attention module (C2C), a feed-forward network, and finally a scene- to-click attention module (S2C). All queries are represented by the positional and the content part. We denote the positional part of all click queries as Q, and the content part at layer E {0, 1, I,..., L} as Q. We use the same representation for the scene points. We denote the point features at layer I as P.. where F = F, which represents the content part of the 3D points. The positional part of 3D points Fp is encoded via Fourier positional encodings (Tancik et al., 2020) based on voxel positions to ensure access to point cloud geometric information to the decoder.
The C2S performs cross- aotssaattnnionn from click queries to point features, which enables click queries to extract information from relevant regions in the point cloud. In the C2C, we let each click query self-attend to each other to realize inter-query communications. The C2C is followed by an FFN that further updates each query. All three steps only update click queries while the point features are static. To make the point features click-aware, we add the S2C that performs cross-attention from point features to click queries. Details on these attention formulations can be found in the Appendix. Query fusion module. We apply a shared mask head (MLP) fmass(-) to convert the per-click content embeddings Q to K mask embeddings and then compute the dot product between these mask embeddings and the refined point features, which produces K mask logits maps C2 R N'X xK
4

---

Published as a conference paper at ICLR 2024
i.e. C. F fmaat((Q') Since there be multiple queries representing the same region (object or background), we apply a per-point max operation on Cç that shares the same region. This step can be achieved through the association between each click and its region index og. and gives us M + region-specifie mask logits map R₂ RN We obtain the final segmentation mask M E 10, M)" N' through so ftmax, which indicates the region each point belongs to.
3.2 USER SIMULATION AND TRAINING
Simulated clicks are commonly used in the interactive community for both training and evaluation. In Kontogianni et al. (2023) positive clicks are uniformly sampled on the object and negative ones are sampled from the object neighborhood. At test time they imitate a user who always clicks at the center of the largest error region. However, the training and test strategies are different since randomly sampled clicks are independent of network errors and lack a specific order. We propose an iterative strategy that approximates real user behavior even during training. Although iterative training has been explored in interactive image segmentation (Mahadevan et al., 2018; Sofiiuk et al., 2022), those methods can only work for single object setup. A detailed comparison with Mahadevan et al. (2018); Sofiiuk et al. (2022) can be found in Tab. 8 and the Appendix.
Multi-object iterative training. Our iterative strategy is shown in Fig. 3. We simulate user clicks for each batch separately in an iterative way with n number of iterations sampled uniformly from Niter. S are the clicks sampled in the i- th iteration. The training starts from the initial clicks (5") col- lected from each target object's center. Full iterative training, like in testing, is costly, requiring an iteration after each sampled click. Therefore, when sampling clicks for the next iteration, in- stead of only sampling one click from the largest error region, we sample N; clicks from the top N, error regions (one click per region). This strategy can generate training samples that contain a large number of clicks in a small number of iterations, keeping the training complexity manageable. We freeze the model when sampling clicks in iterations to Nater - and only allow backpropagation in the last iteration.
Sewerhon
Sever à
Cats
Cuay
Figure 3: Multi-object iterative
training.
Multi-object user simulation during test. Interactive single-object segmentation (Xu al., 2016; Sofiukk et al., 2022: Kontogianni et al., 2023) evaluation strategies imitate a user who clicks at the largest error region. In our multi-object scenario, we share the spirit and enable users focus first on the largest errors in the whole scene. Our user simulation strategy starts with one click at the center of each object to get an initial prediction. We then compute set of error clusters (comparing prediction to ground truth) and sample the next click from the center of the largest cluster.
Loss. We supervise our network with the cross-entropy and the Dice loss (Deng et al., 2018) for multi-class segmentation since we want neighboring masks to compete for space and ensure that each point is assigned to a single label. The number of classes varies per scene and is M + 1, where M is the number of objects the user wants to segment. The total loss is defined as: L I
where λçe and Auise are the balancing loss weights and w, the distance of the points to the user click. Additional implementation details are the Appendix.
N EpEp wp(AccfCc (p) + CDiice
4 EXPERIMENTS
Tasks. AGILE3D a versatile model, able to perform both interactive single- and multi-object 3D segmentation. Since there is no existing benchmark for interactive multi-object segmentation we propose this new task including an evaluation protocol and metrics. We evaluate in both scenarios.
Datasets. A key aspect of interactive segmentation systems is their ability to work on datasets that exhibit significant variations in data distribution compared to the training data. To this end, we train the system on ScanNetV2-Train (Dai et al., 2017), an indoor dataset, and subsequently, we evaluate on datasets that follow distinct distributions, including ScanNetV2 Val (Dai et al., 2017) (same distribution), S3DIS (Armeni et al., 2016) (different sensor), and even KITTI-360 (Liao et al., 2022) (outdoor LiDAR point clouds). We also train two models on ScanNet-train one with 40 classes and another with only the subset of 20 benchmark classes.
5

---

Published as a conference paper at ICLR 2024
Table Quantitative results on interactive single-object segmentation. We compare our method with the current state-of-the-un InterOhject3D ( (Kontogianni et al., 2023) and our enhanced InterOb- ject3D++ in several datasets. Our method offers 3D masks of higher quality with fewer user clicks and generalizes better to new classes and datasets.
Mathod
Train
Eval
Totle51
lot 10
ToUle15
NoC880
NoC885
NoC
InteOOjeet3D
67.6
77.6
81.2
9.5
11.8
14.6
ScmNNe100 Ner4 40 smsso))
76.4
K2.2
83.8
7.8
10.2
11.4
AGILESD (Ours)
78.5
82.9
84.5
9.8
13.1
InterObject3DD
72.4
79.9
82.4
11.2
14.2
IntrOOjeeriDe+ (Oum)
SconNo500 Ner4
78.0
13.2
70.9
ImarOejerT3D
72.4
ScanNee40 -3DDS-AS
80.8
89.2
91.5
6.7
9.3
AGILESD (Our)
83.5
88.2
89.5
6.4
9.5
ImarO0tjcc331)
14.3
26.3
35.0
19.1
19.4
19.7
SemnNer40 KITT1-300
19.9
40.6
55.1
17.0
17.7
18.4
AGILE3D (Ours)
44.4
49.6
54,9
14.2
15.5
16.8
Evaluation metrics. For single-object evaluation we follow the evaluation protocol of Kontogianni al. (2023). We compare the methods on (1) NoC@q% 4. the average number clicks needed to reach 9% IoU, and (2) IoU@k 1, the average IoU for k number of clicks per object (capped at 20). We extend the evaluation protocol for multi-object. We no more enforce a per-object budget but we allow a total budget of M x 20 clicks for user who wants to segment M objects in a scene. We propose the loU@k 1 metric, which represents the average IoU of target objects after an average of k clicks allocated to each object. k is an average of k over M and each object does not necessarily share exactly clicks. Similarly, we report NoC loC@q% 4, which represents the average number of clicks to reach average 9% IoU for all target objects in the scene.
Baseline. We use InterObject3D (Kontogianni et al., 2023) as our baseline in interactive single-object segmentation. However, InterObject3D is designed to segment objects sequentially and cannot directly be evaluated by our interactive multi-object segmentation protocol. To this end, we InterObject3D to pro- cess each object with one click per object sequentially and ob- tain binary mask for each object. We merge those binary masks manually and then use our user simulation protocol to sample the next click. Note this hand-crafted baseline is just for complete comparison and cannot be seen as interactive multi-object seg- mentation. We additionally created strong implementation of InterObject3D, an enhanced baseline (InterObject3D++)) by in- corporating our proposed iterative training. Outperforming this even stronger baseline shows that there is merit in leveraging the clicks of multiple objects together rather than handling objects in isolation.
Figure 4: Open-world segmen- tation from ScanNer20. AGILEE3D segment objects like statue and phone.
4.1 EVALUATION ON SINGLE-OBJEC SEGMENTATION.
Comparison with state-of-the-art. Results are summarized in Tables 1, 2, 3 and Fig 5, 4. We perform the evaluation scenarios of increasing difficulty and distribution shift:
Table Quantitative results interactive single-object segmentation (& 3 clicks), Our method performs signifi- cantly better than baselines, especially in low click regime. Our results for just click comparable better than the clicks of the baselines. Models are trained on SeanNet40.
Table Comparison with fully-supervised. We compare method with the state-of-art fully supervised instance segmenta- tion method. Both methods trained ScanNer20 and evaluated on the ScanNet20-seen and ScanNe220-nsseen.
Method
#elicks
AP
AP sox
AP 25%
Method
IoU@1
loU@2
IoU
Benchmark Classee
Mask3D
51.5
77.0
90.2
SADIS SeanNet
InterObjecr3D
40.8
55.9
63,9
53.5
75.6
91.3
InterObjeet3D++
40.7
60.7
70.2
64.0
86.4
96.0
AGILE3D (Ours)
63.3
70.9
75.4
AGILE3D
3
70.3
91.4
98.1
(Ours)
83.2
98.3
99.8
InterOhject3D
38.5
$4.0
62.5
86.8
99.2
100.0
Int
32.7
55.8
69.0
Mask3D
5.3
13.1
24.7
AGILE3D (Ours)
58.5
70.7
77.4
Enseen Clnsses
KITTI-30
InterObject3D
2.0
5.1
8.5
2
36.9
63.5
85.8
InterObject3De+
3.4
7.0
11.0
AGILE3D
(Ours)
10
67.8
94.8
99.7
AGILE3D (Ours)
34.8
40.7
42.7

---

Published as a conference paper at ICLR 2024
ScanNoo
[oU@p1 83.6
JoU/P2 929
loll@3 94.2
ToU@ 95.3
loU@82 95.8
loU@83 97.0
SaDIS-AS
loU01 61.9
loU@2 = 90.1
loU@3 91.9
ToU@ - 865
loU@22 92.3
loU@33 - 94.5
KITTI-60
Some
loUggS
1o 79.1
1oU@1
loU@3
Ground Truth
1tt
AGILE3D (Our)
Figure 5: Qualitative results on interactive single-object segmentation.
We evaluate on the ScanNet-val dataset, considering small distribu- tion shifts. In both the ScanNet20 and ScanNet40 setups, our method surpasses the baseline and the enhanced baseline, as shown in Tab. 1 1.2). We test the generalization of our method to novel classes by evaluating the trained model on ScanNet 20 classes with the additional 20 unseen classes. Tab. clearly demonstrates that our AGILE3D outperforms the current state-of-the-an in all metrics.
Scannet--miinn-S3DDS We further assess the effectiveness of model's generalization by evalu- ating it on S3DIS, an another indoor dataset with different characteristics than ScanNet. Our model outperforms the state-of-the-an baseline. For example, with only 5 clicks, our AGILE3D achieves an impressive IoU of 83.5 on S3DIS, surpassing the baseline's performance of 72.4.
Our method also excels in the challenging domain shift of training on ScanNet and testing on KITTI-360, an outdoor dataset captured with a LiDAR sensor. Our method surpasses the baseline by factor of 4 and the enhanced bascline by a factor of 2 on IoU@5.
ScamNet--aiin-KI771---
Our method's performance is particularly impressive in the low click regime (< 3 clicks). With just a single click, our method achieves 60 IoU on both ScanNet and the unseen dataset of S3DIS (Tab. 2). It is significant improvement over the baseline of = 40 IoU. With three clicks, our method achieves even higher IoU scores of 75.4 and 77.4 on ScanNet and S3DIS, respectively.
Comparison with fully-supervised methods. Results are summarized in Tab. 3 and Fig. 4. Fully supervised methods for 3D instance segmentation achieve remarkable results on tasks and data dis- tributions similar to those encountered during training. However, we demonstrate that with minimal human feedback, we can surpass the performance fully supervised methods, particularly in classes that were not seen during training. Our method achieves precision 4 times higher than the state-of- the-art method Mask3D (Schult et al., 2023) for unseen classes with just one click (Tab. 3). In Fig. 4, AGILE3D obtains high-quality masks of novel objects (e.g., statue, phone) with few clicks.
4.2 EVALUATION ON MULTI-OBJECT SEGMENTATION.
Table 4: Quantitative results on interactive multi-object segmentation. We adapt the state-of- the-art method in interactive single-object segmentation to be evaluated by our multi-object protocol for a complete comparison (Baseline paragraph of Sec. 4). Note the baselines still predict binary masks for single-object and final masks must be merged manually.
Method
Train Eval
loU@51
loU@10
loU@15
NoC@80
NoC@85
NoC@90 1
InterObject3D
75.1
80.3
81.6
10.2
13.5
16.6
ScanNet40 ScanNet40
79.2
83.3
12.4
AGILE3D (Ours)
82.3
85.0
86.0
10.0
14.4
InterObjeer3D
76.9
85.0
87.3
6,8
8.8
13.5
InterObioctDD++
ScanNett0 S3DIS-A5
AGILE3D (Ours)
86.3
88.3
90.3
9.6
InterObject3D
10.5
22.1
31.0
19.8
19.8
19.9
InterObjectSDett
KITTL.300
16.7
AGILE3D (Ours)
40.5
44.3
48.2
17.4
18.3
18.8
Results are summarized in Tab. 4 and Fig. 6, 7. More qualitative results in Appendix. We adapted InterObject3D for our multi-object protocol for a complete comparison. We do not enforce per- object click budget for the baselines but allow them sample the next click from the biggest error region across all target objects. Nevertheless, the baselines are still limited to segmenting one object in each forward pass. AGILE3D outperforms all the baselines, requiring significantly fewer clicks

---

Published as a conference paper at ICLR 2024
Chair
Chair
Chuir
Chuir
IntaOkjjertD++
Merged
(a) 1oU - 77.0
(b) lolig? 75.9
(c) 1oU 8133
(4) Iol 71.6
let laUge 76.0
AGILESD
(Ohus)
(0 10D0SS 74.1
(g) lollees
(8)
( ToU
(1 Tollee 83.4
Qualitative results interactive multi-object segmentation ScanNet40. user wants to segment chairs using InterObject3D++ (top) AGILE3D (botiom). (a)-(d) show the results for each object and (e) shows the final merged prediction after a total of clicks for InterObjecl3D++, (f-(i) show the results of AGILE3D after each iteration. New click in each iteration is marked with a red circle.
Figure 6:
Maraed
Merged
3D Scone
lo0l0s 90.4 loU@! 565 IMP? 817 lolgs 75.8 rathas 2019
3D Sorne
100004 = 961
IoU 63.2
GOR8 787
Siliah 99.2
InterObjecL1
AGILEJD (Ours)
InterObject3
AGILE3D (Ouss
Qualitative comparison S3DIS-AS
(b) Qualitative comparison KITTI-360
Figure 7: More qualitative comparison. AGILE3D achieves higher IoU, e.g., when segmenting cabinets and boxes S3DIS-A5 (lefi) and pedestrians on KITTI-360 (right).
for the same quality of masks, e.g., 4 clicks less than InterObject3D and clicks less than Inter- Object3D++ to achieve on average 80% loU on ScanNet40 (Tab. 4, In 1-3). The benefits of multi- object handling in AGILE3D are also validated qualitatively in Fig. In this scene, after a total of 8 clicks (both methods), AGILE3D achieves an average IoU of 83.4 vs. 76.0 of InterObject3D+++ These results highlight the benefits of interactive multi-object segmentation: (1) Click sharing: in AGILE3D, clicks on object naturally utilized segment other objects, e.g., the positive click on Chair 1 (Fig. 6-g) naturally serves as a negative click for Chair 2 and improves the segmentation for both objects (compare with Fig. 6-f). By contrast, in the baselines, clicks indi- vidually given and only have an effect on that one object. Please note simply equipping the baselines with click sharing does not bring performance gains (see Appendix). (2) Holistic reasoning: since we segment all the objects together, AGILE3D can capture their contextual relationships, enabling holistic reasoning. For example, clicks on the armrest of one chair help correct the armrests of other chairs (Fig. 6-i.)). (3) Globally-consistent mask: AGILE3D encourages different regions to directly compete for space in the whole scene so that each point is assigned exactly one label. By contrast, in single-object segmentation, the mask for each object is obtained separately. Post-processing is required to generate non-ov overlappag masks. (4) Faster inference: AGILE3D pre-computes back- bone features once per scene (~~0.05s) and runs light-weigh decoder per iteration (~~0.02s). By contrast, InterObjectD++ goes through the entire network per iteration (~~0.55s). In this scene, after 8 clicks, AGILE3D has inference time of 0.5 vs. 0.4s of InterObject3DD++
Efficiency Comparison shown Tab. 5. The inference time is measured after on average 5/10/15 clicks allocated to each object in each scene. The results show that our model has comparable memory consumption but our inference time (which is much more critical for real-time applications) 2x faster than the baselines due to our model architecture design (Fig. 1).We report FLOPs for completeness however the results are not comparable since the MinkowskiEngine uses some clever hashing operations that can not be measured by existing FLOPs profilers.
4.3
USER STUDY
To go beyond simulated user clicks and assess the performance with real human clicks, we perform a user study. Tab. demonstrates that real users achieve results comparable to the simulator: The results also verify that iterative training strategy improve the performance of both the real user and the simulator. Morcover, the variance indicates that the model trained with iterative training more robust across different user behaviors.
In Tab. 7, we compare the results of real users in single-object and multi-object settings. We report the total given clicks per object, the achieved IoU per object, and the time spent per scene. In multi- object settings, users achieved higher loU with significantly fewer clicks in less time, indicating that the multi-object setting be more beneficial. Details the user study design in the Appendix.
8

---

Published as a conference paper at ICLR 2024
Table 5: Efficiency comparison. Our
Table 6: User study on itera- tive vs random sampling.
Table 7: User study on annotation of 30 objects.
method is 2x faster than the baselines.
Method
FLOPuG
ManuMb
1955
te 3D
10:35
Car
Training Inile
var.
NoZ
van
Settings
NoC ToU i
InterOge::131
1.58
924
1.2
2,4
3.6
Human Deratioe
96.8
So-5
InterO
1.58
924
24
3,5
Sim.
88.2
2.1
Single-object 12.6 87.4 4min
AGILESD
16.57
710
1,0
1.6
Random
85.5
6e-4
0,1
Multi-object 5.8 88.0 3min
4.4 ABLATION STUDIES AND DISCUSSION
We ablate several aspects of our architecture and training in Tab. 8, 9. All the experiments are con- ducted on the ScanNet40 dataset on multi-object scenario unless stated otherwise. More ablations are available in the Appendix.
Iterative training. We validate the effectiveness of our iterative training strategy (Sec. 3.2) by training model with randomly sam- pled clicks. Tab. 9, ① shows that the random sampling strategy performs noticeably worse than our iterative training (82.0 vs. 84.4
Table 8: Iterative training strategies (single-object).
Method
IoU@10
NoC@s5
loU@ 10). We also compare with two popular iterative strategies: ITIS (Mahadevan et al., 2018) and RITM (Sofiiuk et al., 2022). We compare the single-object setting for a fair comparison since they designed for such a setting. Tab. 8 shows that the model trained using our proposed training strat- egy significantly outperforms models trained using ITIS or RITM. More details in the Appendix.
ITIS
79.9
11.1
RITM
81.4
10.3
AGILE3D
82.9
9.8
Attention design. We design click attention module to enable ex- plicit interaction between the click queries themselves and between them and the point features. Tab. 9 shows that the absence of any type attention mechanism harms the model's performance. Espe- cially, the cross-attention between click queries and point features, Le.. @ C2S attn and ( S2C attn, have a more pronounced effect compared the self-attention between click queries ④ C2C attn. We note that C2C attn has significant effect on the transfer exper- iments for KITTI-360, e.g. 37.4 (with C2C) vs. 33.8 (wo. C2C) on loU @5 indicating that C2C plays a crucial role for improving the model's generalization ability.
Table 9: Ablation study.
Methods
totre
Nccoss
(Uers)
ismmiie toining
12.2
C2S
$2.8
12.0
C2C um
$4.0
10.8
S2C utaa
13.1
11.4
all ancaatoo
79.2
13.7
spatial port
10.8
Somporal
1088
both parts
11.2
Spatial-temporal encodings for click queries. We regard user clicks as an ordered sequence of 3D coordinates and supplement each click query with a spatial-temporal en- coding. Tab. 9, 7 demonstrates that both the spatial and temporal encoding contribute to improved performance. Additionally, KITTI-360, we note the effect of spatial- temporal encodings is even more pronounced, e.g. remov- ing the temporal part led to drop from 42.3 to 40.5 and removing the spatial part led to a drop to 36.9 on IoU@ 10.
3D scese
(b) diik
3 click
(4) 15 clicks
Figure 8: Failure cases fine- ine-grained segmentation, e.g. part segmentation of chair.
Limitations and failure cases. When users click on the seat of chair, do they want segment the entire chair or only the seat? Our model might not handle such ambiguity explicitly. Although AGILE3D can segment objects in its entirety very well (often with a single click as shown in Fig. b), it may struggle when segmenting object parts (Fig. 8 c) and requires more clicks to segment object parts correctly (Fig. 8 d). This can be potentially solved by learning object-part relations and we leave it as future work. Second, like all interactive segmentation models, AGILE3D does not support yet a prediction of semantic labels along with the predicted mask. Equipping interactive segmentation models with semantic-awarenes, could be a potential future work.
5 CONCLUSION
We have proposed AGILE3D, the first interactive 3D segmentation model that simultaneously seg- ments multiple objects in context. This is beyond the capabilities of existing interactive segmenta- tion approaches, which limited to segmenting single objects one by one. While offering faster inference, AGILE3D further achieves state-of-thhe-ar in both interactive single- and multi-object benchmarks, in particular in the challenging low-elick regime. We believe AGILE3D opens new door for interactive multi-object 3D segmentation and can inspire further research along this line.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We sincerely thank all volunteers who participated our user study. Francis Engelmann and Theodora Kontogianni are postdoctoral research fellows at the ETH Al Center. This project is partially funded by the ETH Career Seed Award Towards Open-World 3D Scene Understand- ing, NeuroSys-D (03ZU1 BMBF projects 6GEM (16KISK036K) and Hasler Stiftung Grant Project (23069).
REFERENCES
Eirikur Agustsson, Jasper RR Uijlings, and Vittorio Ferrari. Interactive Full Image Segmentation by Considering All Regions Jointly. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3D Semantic Parsing of Large-Scall Indoor Spaces. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitScenes A Diverse Real- World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-Scale Interactive Object Segmentation with Human Annotators. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and Xinggang Wang. Hierarchical Aggregation for 3D Instance Segmentation. In International Conference on Computer Vision (ICCV), 2021.
Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. FocalClick: Towards Practical Interactive Image Segmentation. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- Attention Mask Transformer for Universal Image Segmentation. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Julian Chibane, Francis Engelmann, Tuan Anh Tran, and Gerard Pons-Moll. Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation using Bounding Boxes. In European Conference on Computer Vision (ECCV), 2022.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4D Spatio-Temporal Convnets: Minkowski Convolutional Neural Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Angela Dai, Angel X Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes. In IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), 2017.
Ruoxi Deng, Chunhua Shen, Shengjun Liu, Huibing Wang, and Xinru Liu. Learning to Predict Crisp Boundaries. In European Conference on Computer Vision (ECCV), 2018.
Fei Du, Jianlong Yuan, Zhibin Wang, and Fan Wang. Efficient Mask Correction for Click-Based In- teractive Image Segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2023.
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias Nießner. 3D-MPA: Multi-Proposal Aggregation for 3D Semantic Instance Segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep Learning for 3D Point Clouds: Survey. Transactions on Pattern Analysis and Machine Intelli- gence (PAMI), 2020.
10

---

Published as a conference paper at ICLR 2024
Yong He, Hongshan Yu, Xiaoyan Liu, Zhengeng Yang. Wei Sun, Yaonan Wang, Qiang Fu, Yan- mei Zou, and Ajmal Mian. Deep Learning Based 3D Segmentation: A Survey. arXiv preprint arXhv:2103.05423, 2021.
Ji Hou, Angela Dai, and Matthias Nießner. 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring Data-Efficient 3D Scene Understanding With Contrastive Scene Contexts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Won-Dong Jang and Chang-Su Kim. Interactive Image Segmentation via Backpropagating Refine- ment Scheme. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Theodora Kontogianni, Michael Gygli, Jasper Uijlings, and Vittorio Ferrari. Continuous Adaptation for Interactive Object Segmentation by Learning From Corrections. In European Conference on Computer Vision (ECCV), 2020.
Theodora Kontogianni, Ekin Celikkan, Siyu Tang. and Konrad Schindler. Interactive Object Seg- mentation 3D Point Clouds. In International Conference on Robotics and Automation (ICRA), 2023.
Zhuwen Li, Qifeng Chen, and Vladlen Koltun. Interactive Image Segmentation with Latent Diver- sity. In JEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2022.
Qin Liu, Meng Zheng, Benjamin Planche, Srikrishna Karanam, Terrence Chen, Mare Niethammer, and Ziyan Wu. PseudoClick: Interactive image segmentation with click imitation. European Conference on Computer Vision (ECCV), 2022.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Con- ference Learning Representations (ICLR), 2019.
Sabarinath Mahadevan, Paul Voigtlaender, and Bastian Leibe. Iteratively Trained Interactive Seg- mentation. In British Machine Vision Conference (BMVC), 2018.
Amit Kumar Rana, Sabarinath Mahadevan, Alexander Hermans, and Bastian Leibe. DynaMITe: Dynamic Query Bootstrapping for Multi-object Interactive Segmentation Transformer. In Inter- national Conference Computer Vision (ICCV), 2023.
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D for 3D Semantic Instance Segmentation. In International Conference on Robotics and Automation (ICRA), 2023.
Tianchang Shen, Jun Gao, Amlan Kar, and Sanja Fidler. Interactive Annotation of 3D Object Ge- ometry Using 2D Scribbles. In European Conference on Computer Vision (ECCV), 2020.
Konstantin Sofiiuk, Ilya A Petrov, and Anton Konushin. Reviving Iterative Training with Mask Guidance for Interactive Segmentation. International Conference on Image Processing (ICIP), 2022.
Jiahao Sun, Chunmei Qing, Junpeng Tan, and Xiangmin Xu. SuperPoint Transformer for 3D Scene Instance Segmentation. AAAI, 2023.
11

---

Published as a conference paper at ICLR 2024
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kar Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießnnr, Antonio Criminisi, Shahram Izadi, and Philip Torr. SemanticPaint: Interactive 3D Labeling and Learning at Your Fingertips. ACM Transactions on Graphics (TOG), 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention All You Need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo. SoftGroup for 3D In- stance Segmentation on Point Clouds. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Qiaoqiao Wei, Hui Zhang, and Jun-Hai Yong. Focused and Collaborative Feedback Integration for Interactive Image Segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Binbin Xiang, Yuanwen Yue, Torben Peters, and Konrad Schindler. A Review of Panoptic Segmen- tation for Mobile Mapping Point Clouds. ISPRS Journal Photogrammetrr and Remote Sensing, 2023.
Ning Xu, Brian Price. Scott Cohen, Jimei Yang, and Thomas S Huang. Deep Interactive Object Selection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas Guibas. GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Shuaifeng Zhi, Edgar Sucar, Andre Mouton, lain Haughton, Tristan Laidlow, and Andrew Davison. iLabel: Revealing Objects in Neural Fields. IEEE Robotics and Automation Letters, 2022.
Minghao Zhou, Hong Wang, Qian Zhao, Yuexiang Li, Yawen Huang, Deyu Meng, and Yefeng Zheng. Interactive Segmentation As Gaussion Process Classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Qian-Yi Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018.
Appendix
The appendix is organized follows:
SA: Additional model details including:
Architecture and detailed formulations of our click attention module. Implementation details of the whole model and training procedure.
BB More experimental results including:
More qualitative results in various challenging cases.
Labeling new datasets without ground truth for ARKitScenes.
More qualitative results KITTI-360, the outdoor LiDAR sean dataset.
Additional ablation study on the design choices of our query fusion strategies.
Additional ablation study on the term balance in our spatial-temporal query encoding.
Theoretical and experimental comparison of our proposed iterative training strategy with ITIS (Mahadevan et al., 2018) and RITM (Sofiiuk al., 2022).
12

---

Published as a conference paper at ICLR 2024
Additional ablation study on extending baselines (InterObject3D and InterObject3D++) for click sharing.
More analysis on backbone feature maps and click query attention maps.
C:: Additional details about user studies including study designs and our developed user interface.
SD: Details of datasets and the data preparation procedure for our interactive multi-object seg- mentation setup.
A MODEL DETAILS
A.1 CLICK ATTENTION FORMULATIONS
The detailed architecture of our click attention module is shown in Fig. 9. All queries are represented by the positional part Qp and the content part Q which are initialized through our module. We use the same representation for each point in the 3D scene. The content part of the scene F! is initialized from backbone features and the positional part is obtained via Fourier positional encodings (Tancik al., 2020). First, click-to-scent attention module performs cross-attention from click queries to point features, which enables click queries to extract information from relevant regions in the point cloud (Eq. 1). In practice, we employ masked attention (Cheng et al., 2022), which applies attention mask H to constrain the attention of each click query to the points within its intermediate predicted mask from the previous layer. Then in the click-to-clice attention module, each click query self-attends to each other to realize inter-query communications (Eq. 2). To make the point features click-aware, we let the point features cross-attend to the click queries a scene-to- click attention module (Eq.3), In equations (1) to (3), omit the layer normalization and dropout for simplicity. We. WK. Wy are learnable weights for query, key and value as in the standard attention mechanism (Vaswani et al., 2017). In all attention modules, we add the positional part to their respective keys/queries.
WolC' Qp) WK + Fp) F2)++H + WV Q' VD
Q++1 softmax
(1)
Wolo! + 2p) W WK (Q: 1rr)
Q++ softmax
Wrv + Q
(2)
VD
(WolF++ + Fp)
Wx Wx(Q! Qp) VD
F++1 softmax
..r
(3)
A.2 IMPLEMENTATION DETAILS
Model setting. We use the same Minkowski Res16UNe (Choy et al., 2019) backbone as Kon- togianii et al. (2023); Schult et al. (2023). We first quantize the 3D scene into N' voxels with fixed size of 5cm as Kontogianni et al. (2023); Schult et al. (2023). The backbone takes as input the sparse voxelized 3D scan and produces feature map F E 1R x96 The feature maps are further projected to 128 channels by 1xl convolution. The click attention module consists of layers with 128 channels. All attention modules in each layer have 8 heads. The feed-forward network has feature dimension of 1024. In addition to regular background click queries generated from users' given background clicks, we also additionally use 10 learnable background queries, which are omitted in the architecture figure (Fig. 2) for visual clarity.
and the ADize 2 in the loss function. The loss is applied to every intermediate layer of the click attention module. We use the AdamW optimizer (Loshchilov & Hutter, 2019) with weight decay factor le-4. We train the model on ScanNet40 for 1100 epochs with an initial learning rate le-4, which is decayed by 0.1 after 1000 epochs. Due to the smaller data size, train the model on ScanNet20 for 850 epochs with an initial learning rate le-4, which decayed by 0.1 after 800 epochs. We use a single TITAN RTX GPU with 24GB memory for training.
Training. We set the ÀCE
13

---

Published as a conference paper at ICLR 2024
1
Maxl Head
Click
Attention
Module
Clecko 2o Clesk ABE.
Initial Click
Click Mask
Region Mask
Querles
Legits
Logits
(a) Late fusion
Cleb-ttSSerre Auns
Layer
Attention Module
02
Qp
mean
Click
Region
Quertis
Querles
Logits
3D Secric
Clicks
(b) Early fusion
Figure 9: Structure of the click attention module. We omit the FFN block clarity.
Figure 10: Different fusion strategies. (a) Late fusion and (b) Early fusion
B ADDITIONAL RESULTS
B. MORE QUALITATIVE RESULTS IN VARIOUS CHIALLENGING CASES.
We include more qualitative results of interactive multi-object 3D segmentation in various chal- lenging cases in Fig. 11, e.g., target objects that are occluded by other objects, small objects, con- nected/overlapped objects, un-rigid objects, thin structures around objects, and extremely sparse outdoor scans.
B.2 LABELING NEW DATASETS WITHOUT GROUND TRUTH FOR ARKITSCENES
In the main paper, conducted extensive evaluations real-world datasets ScanNet (Dai al., 2017), S3DIS (Armeni et al., 2016) and KITTI-360 (Liao et al., 2022). To further demonstrate the practical usage of work, scenes from ARKitScenes (Baruch et al., 2021), which is diverse real-world dataset captured with the Apple LiDAR scanner but does not contain annotations for instance masks. Here show our generated segmentation results in Fig. 12. Although our model is only trained on ScanNet, it can generalize well to unseen datasets, which is a crucial ability to label new datasets in practice. Typically with only about I click on average per object (Second column), our model can already produce visually satisfying results. With more clicks, our model can further refine the segmentation and produce higher-quality masks (Third column).
interactive tool to annotate variety of challenging
B.3 MORE QUALITATIVE RESULTS ON KITTI-360
We show more qualitative results our method in multi-object setting on KITTI-360 in Fig. KITTI-360 (Liao et al., 2022) has the largest domain gap with ScanNet40 since it is an outdoor dataset and the point cloud is much sparser. To push the limits of our model, we additionally eval- uate our model on the single scan, which is even sparser. Our model is effective for multi-object segmentation on both accumulated scans and single scans. Please note we only train our model on the indoor dataset ScanNet and directly evaluate the outdoor dataset without any fine-tuning. Those results again demonstrate the strong generalization ability of our model on datasets with large domain shifts.
14

---

Published as a conference paper at ICLR 2024
(a) Occluded objects
3D Scan
Ground Truth
IoU@1 86.3
IoU@2 2 88.5
(b) Small objects
3D Scan
Ground Truth
loU@T I 71.6
loU@ī 76.2
(c) Overlapped objects
3D Scan
Ground Truth
IoU@ 71.6
IoU@7 = 80.0
(d) Un-rigid objects
3D Scan
Ground Truth
IoU@ T 52.6
ToU@3 = 74.4
(e) Thin structures
around objects
Scan
Ground Truth
lou@1 63.4
loU@5 = 73.0
(f) Extremely sparse scans
3D Scan
Ground Truth
loU@3 62.2
loU@10 90.4
Figure 1I: More results on challenging cases interactive multi-object segmentation. AG- ILE3D is robust across various challenging scenarios, e-g., (a) table is largely occluded by boxes and shelf. (b) small objects like bottles and jars. (c) three pillows are connected and overlap each other. (d) un-rigid objects such as curtains and bathrobes. (e) Very thin structures like door frames around doors. (f) extremely sparse scans in outdoor scenes. Best viewed color and zoom in.
15

---

Published as a conference paper at ICLR 2024
click/object
2.7 elick/object
click/object
3.2 click/object
click/object
click/object
1.25 click/object
4 clickfobject
click/object
click/object
click/object
click/object
Figure 12: Labeling new scenes for ARKitScenes (Baruch et al., 2021). Since ARKitScenes does contain ground truth instance masks, we show qualitative results annotated using our interactive tool. Although our model is only trained on ScanNet (Dai et al., 2017), it can be effectively used to annotate unseen datasets. Typically with only about click per object, our model can already gen- erate visually satisfying results (Second column). More clicks further improved the segmentation. The number of clicks is averaged over all target objects.
B.4 ABLATION ON QUERY FUSION STRATEGY
In our query fusion module, we apply a per-point max operation to aggregate tlick-specffi masks to region-specific masks (Fig. 10 a). To validate this design, we compare with early fusion strategy
16

---

Published as a conference paper at ICLR 2024
Accumulated
Truth
I0U-16.7
Ground Trush
Single Scans
1-2
16U-17.3
Ground Truth
IoU-333
Ground Truth
Figure 13: Qualitative results on interactive multi-object segmentation on outdoor scans. We evaluate AGILE3D both accumulated scans and single scans. Even though AGILE is only trained on the indoor dataset ScanNet40, it can effectively segment outdoor seans (either accumulated or single scans).
Table 10: Ablation on query fusion.
Table 11: Ablation on balance in spatial-temporal query encoding.
Methods
lou@
NoC@85
Temporal
IoU TO
NoC@
fusion max (ours)
10.7
Late fusion- mean
83.8
10.9
83.9
11.1
1:2
83.6
11.2
Early fusion max Early fusion- mean
13.6
2:1
83.7
11.2
13.4
Concatenation cotiitennttioo-Poocctiiii
83.7
11.3
where we aggregate all initial click queries that share the same region into a single query before feeding them to click attention module (Fig. 10 b). For both fusion strategies, investigate both max and mean aggregation. Tab. 10 shows the late fusion strategy outperforms the early fu- sion strategy significantly. Although the early fusion strategy can reduce the query number and is more computationally efficient, it may cause information loss. By contrast, the late fusion strategy encourages each query learn represent different parts of the target object (Fig. 15). The final fusion of these queries produces strong and global representation of the target object.
B.5 ABLATION ON TERM BALANCE IN SPATIAL- ENCODING
In our click-as-query module, we consolidate the spatial and temporal parts to positional part by direct summation. We show the ablation on the weights of the spatial and temporal part in the summation in Tab. 11. In addition to the weighted sum, also try to concatenate the spatial and temporal parts into feature vector then followed by linear projection layer. The results show method works well with different weights. The weighted sum has similar scores with
17

---

Published as a conference paper at ICLR 2024
the concalenaation-proectionn strategy but is more efficient without introducing additional model parameters. We choose 1:1 in the main paper to avoid extensive hyperparameter tuning.
B.6 COMPARISON OF ITERATIVE TRAINING STRATEGIES
Table 12: Comparison of iterative training strategies.
Method
IoU@5t t IoU@ 10
NoC@so IoU@15
NoC@85 i NoC@90 |
ITIS
74.7
79.9
81.9
8.7
11.1
14.2
RITM
77.7
81.4
82.9
7.9
10.3
13.5
AGILE3D
78.5
82.9
84.5
7.4
9.8
13.1
While acknowledging the previous exploration of iterative training interactive 2D segmenta- tion (Mahadevan et al., 2018; Sofiiuk et al., 2022), underline two key distinctions in our approach: (1) Multi-object in 3D setup: Adapting the iterative training strategy directly from single-object 2D segmentation to our context is not possible. Unlike simple click sampling from colse-positives or false-negativer our strategy addresses eπror regions of multiple objects holistically. (2) Fully it- erative sampling: Prior approaches use random sampling as initialization complexity and then incorporate a few iterative samples. In contrast, our training follows a fully it- crative process. We start with one click per object and then sample clicks from the top error regions (one per region) in each iteration. Our method generates numerous training samples/clicks in a few iterations. Thus, it maintains reasonable training complexity while enhancing performance.
avoid computational
Here, experimentally compare with previous training strategies: ITIS (Mahadevan et al., 2018) and RITM (Sofiiuk et al., 2022). The training strategy in RITM built upon ITIS with improve- ments, c.g., batch-wise iterative training instead of epoch-wise iterative training. However, both methods are designed for interactive single-object image segmentation. For comparison, we adopt ITIS and RITM train model in single-object setting and compare with our model trained using our training strategy. The results on ScanNet40 are shown in Tab. 12.
The model trained using our proposed training strategy significantly outperforms models trained using ITIS or RITM. Please note we evaluate the model in single-object setting, which already favors ITIS and RITM since they are designed for that. The results demonstrate the superiority of our proposed iterative multi-object training scheme.
B.7 EQUIPPING THE BASELINE WITH CLICK SHARING ABILITY
Table 13: Additional ablations click-sharing for the baselines ScanNet40 dataset
Method
loU@
loll@ f
InU@
NoC@80
NoCW8
NoC699
InterObject3D
75.1
80.3
10.2
13.5
16.6
InterObject3D click sharing
73.7
80.0
81.2
10.7
13.7
16.7
InterObjeci3D++
79.2
82.6
83.3
8.6
12.4
15.7
OhterSDjectDD+++ click sharing
79.4
83.0
83.7
8.6
12.2
15.5
AGILE3D (Ours)
82.3
85.0
86.0
6.3
10.0
14.4
It is natural to ask whether we can simply equip the baseline with the click-sharing ability. We tried to adapt the baseline to evaluated by our interactive multi-object segmentation protocol for fair comparison in Tab. 4. Now we further modify the baseline to support click sharing: we save the positive clicks of already segmented objects and add those clicks the negative clicks for the next object. We report the new evaluation results on ScanNet40 in Tab. 13.
The results show that this extension only brings marginal gains to InterObject3D+++ and even harms the performance of the original InterObject3D. We speculate that this is due to limitations inherent in the single-object network design and training process:
(1) The baseline models encode clicks two clicks maps: one for positive and one for negative clicks. During training, the models only see clicks related to a single object and typically those clicks are spatially close. However, if we directly equip the single-object model with click-sharing during inference, the model encounters "unrelated" clicks from other objects. This would introduce different distribution the clicks maps from those during training.
18

---

Published as a conference paper at ICLR 2024
(2) Even though equipped with click-sharing, the baseline model still only predicts binary mask and the masks of different objects have no direct communication. contrast, our AGILE3D is an end-to-end model that directly predicts the multi-object masks in holistic manner.
B.8 MORE ANALYSIS ON BACKBONE FEATURE MAPS AND CLICK QUERY ATTENTION MAPS
What does the backbone learn? As we only input the 3D scene to the backbone, the backbone learns general object-level features which we visualize Fig. These features are beneficial for click queries for extracting target object features in the click-to-scene attention module.
What does each click query attend to? We encode clicks as queries, which undergo iterative up- dates by cross- cross-attending to point features and self-attending to one another, resulting in meaningful representation of the target object. In Fig. 15, we visualize the attention maps cach click query which reveal that each query attends to specific regions, e.g., click e₁ attends to the entire chair with emphasis on legs, C2 captures the general shape of the table while Cs focuses on the nearby leg, aligning well with the user's intention to click og for refining the segmentation of the table leg.
Figure 14: Learned backbone features. AGILE3D learns general object-level features. The point features are projected to RGB space using principal component analysis (PCA).
C2
C1
Predicted mask
Click C₁
Click C2
Click C3
Figure 15: Attention maps for click query C1, C c3, respectively. Each query attends to specific regions.
C USER STUDY
Design, To go beyond simulated user clicks and assess performance with clicks from real human user behavior, we perform real user studies. To that end, first implement interactive annotation tool with user-friendly interface (Fig. 16). Our users were not professional labelers and they have used such tool before. We showed them written instructions and also explained verbally to them how the tool works. We did not explicitly instruct the user to click the overall max eттor region but instead allowed them to follow their preferences. Before recording their results, we allow the users to label an example scene to familiarize themselves with the tool. Each user is asked to annotate 5 scenes, each which consists of 4-7 objects. Those objects cover a variety categories, including chairs, tables, beds, lamps, telephones, etc. Please note all users labeled the same random objects for comparable results. We conducted two user studies:
(1) 12 real iterative training strategy (Sec. 3.2 in the main paper), and the other users use our model trained with random sampling.
split into groups, where 6 users
model trained with proposed
(2) 12 real users are split into two groups, where users annotate objects one by one in a single- object setting, and the other 6 users annotate objects in our proposed multi-object setting.
The results and analysis can be found in Sec. 4.3 in the main paper.
User interface. Our user interface is shown in Fig. 16 and developed based on the library Open3D (Zhou et al., 2018). The software can run across platforms and on browsers, and sup- ports both interactive single- and multi-object segmentation. We develop various keyboard shortcuts
19

---

Published as a conference paper at ICLR 2024
Figure 16: User interface
ease user interaction, e.g., Ctrl + Click would identify the background, and Number + Click would identify object. We will release the source code of AGILE3D as well as the annotation tool to facilitate future research.
D BENCHMARK DETAILS
D.I DATASET
Vc (Dai et al., 2017) is richly-annotated dataset of 3D indoor scenes, covering diverse room types such as offices, hotels, and living rooms. It contains 1202 scenes for training and 312 scenes for validation well as 100 hidden test scenes. ScanNetV2 contains the segmentation mask for 40 classes, referred to as ScanNet40. However, the official benchmark evaluates only on 20- class subset, referred to ScanNet20. To test the model's generalization ability, we also the 20-class subset training set to train the model and evaluate on both the benchmark 20 classes and the remaining unseen 20 classes.
S3DIS (Armeni et al., 2016) is a large-scale indoor dataset covering six areas from three campus buildings. It contains 272 scans annotated with semantic instance masks of 13 object categories. Following Kontogianni et al. (2023), we evaluate on the commonly used benchmark split ("Area test").
KITTI-360 (Liao et al., 2022) is large-scale outdoor driving dataset with 100k laser scans in driving distance of 73.7km. annotated with dense semantic and instance labels for both 3D point clouds and 2D images. The dataset consists of 11 individual sequences, each of which corresponds to continuous driving trajectory. We evaluate the task interactive segmentation on the sequence 2013 05 28 drive.00i sync. sync.
D.2 DATA PREPARATION
In our interactive multi-object benchmark, we aim to segment multiple objects in scene simultane- ously. However, 3D scene may contain large number of objects. As shown in Fig. 17, most of the scenes in ScanNetV2 (Dai et al., 2017) training set contain 20-40 objects and several scenes even contain more than 100 objects. Moreover, S3DIS (Armeni et al., 2016) contains few very large spaces, e.g., hallways with even more than millions of points. In practice, would not be feasible for user to simultaneously annotate all the objects in large scene. Motivated by this considera-
20

---

Published as a conference paper at ICLR 2024
tion, set the maximum number of target objects in each evaluation sample 10. For each scene, we randomly select E [1, 2, 10] nearby objects as evaluation samples. Since such selection involves randomness, plan to release the object ids for a fair comparison with our method.
sum. of
20
15
10
20
60
80
100
120
Num.
Object
Figure 17: Object counts in ScanNet training set
21


Reviews:
[{"review_id": "2AjtLSEzgr", "reviewer": "Reviewer_byjT", "summary": "The paper presents AGILE3D, the interactive 3D segmentation model that simultaneously segments multiple objects in context. The key idea is to use the attention-guided interactive 3D segmentation approach and also propose the click attention module to build the connection between different clicks, finally achieving multi-object segmentation via fast inference. The experimental results demonstrate and validate the proposed achievement of better performance on ScanNetV2 and outperform the competitive method InterObject3D.", "questions": "The strengths of the paper lie in the comprehensive information provided, the inclusion of supplementary materials, and the thorough explanations. However, the lack of novelty, limited evaluation, and other weak issues Although the appendix serves its purpose as a resource for implementing AGILE3D, it does not significantly contribute to the field. Considering these strengths and weaknesses, I am negative about the submission currently, but I look forward to the response to the above questions.\n\nsee weakness", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The paper provides detailed information and resources about AGILE3D, allowing readers to gain a deeper understanding of the framework and its implementation.\n2) The appendix includes additional materials such as network architecture, algorithms, and data samples, which further support the understanding and practical implementation of AGILE3D. The authors provide clear and detailed explanations of the concepts, algorithms, and techniques used in AGILE3D, ensuring that readers can follow and reproduce the framework effectively.", "weaknesses": "1) Lack of Novelty: For the proposed methods, the novel point is the multi-object segmentation, and the fast inference is also a novel point to some degree. And for efficiency, the submission does not present a technical point on how to improve efficiency. Multi-object segmentation is also achieved by the previous methods (InterObject3D) via some small improvements. Actually, the performance on the multi-object segmentation is not the best, according to Table 4.\n2) Limited Evaluation: The appendix focuses more on providing information and supplementary materials than conducting extensive evaluations. A more thorough evaluation of the framework's performance and comparison with existing approaches would enhance the paper's credibility. The various datasets are also a strong evaluation, especially for the interactive tools. I suggest some results on the real captured and other challenging datasets should be presented in the paper.\n3) The running time should be reported in a fair manner; for table 5, it is not clear how to perform the comparison since the manuscript claims efficiency as a contribution. Or, the user study is a good choice to compare the efficiency.\nIn some challenging cases, there are a lot of occlusions between the segmented object and other objects in the scene.\n4) The significant difference between interobject3D and interobject3D++ should be discussed comprehensively since the published paper is very relative to the submission.\n5) From Figure 1, the input only presents the XYZ + RGB, i.e., the point cloud and point color. Why does the visualization have the faces? Could you use the faces as extra information to enhance the performance of the proposed methods? From the interactive demo, there is still the face visualization.\n6) The related works are very sparse, especially for the 3D instantce segmentations. I suggest the author refer to some surveys about the 3D segmentation and discuss more papers on it, especially for the pointnet and pointnet++.\n7) For the comparison, it is not clear to me: Are the fewer user clicks the same for the different methods?\n8) The failure cases and limitations are not discussed comprehensively in the submission; these parts are very important for the complete manuscripts, and some failure cases should be presented in the submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ri92Zh1FAR", "reviewer": "Reviewer_PRcH", "summary": "The paper introduces AGILE3D, an attention-guided interactive model for multi-object 3D segmentation. At its core, AGILE3D encodes user interactions as spatial-temporal queries, leveraging a unique click attention module to foster interplay between user-generated queries and the 3D scene. The model not only supports concurrent segmentation of multiple 3D entities but also champions efficiency, demanding fewer user inputs and providing swifter inference times. Comprehensive evaluations indicate its superiority over extant methods across both individual and multi-object interactive segmentation benchmarks.", "questions": "Please check the suggestions in the Weakness section above to further strengthen the paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The proposed approach pioneers in the realm of interactive multi-object 3D segmentation by introducing an avant-garde attention-centric model.\n- The real-world practicality of the proposed approach is convincing with achieving top-tier results on both the popular individual and multi-object interactive segmentation benchmarks, w.r.t. SOTA methods such as Mask3D. I also appreciate the detailed computational cost comparisons.\n- AGILE3D's potential to discern multiple entities with diminished user interaction and expedited inference, vis-à-vis its counterparts, is empirically substantiated.\n- The authors offer a cogent review of pertinent literature, spanning interactive 3D and image segmentation domains, accentuating their contribution's novelty.\n- The user study with a proper interactive interface provides convincing signals for the superiority of the model.", "weaknesses": "- Discussion of limitations/drawbacks should be put in the main body of the manuscript instead of the appendix for a fairer portrayal.\n- More failure cases need to be showcased and analyzed if any, otherwise the paper seems to focus too much on the advantages of the presented method and does not always give the whole picture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wezzQrRlqj", "reviewer": "Reviewer_3vqw", "summary": "The paper proposes a new model for interactive 3D object segmentation called AGILE3D, which firstly encodes points to features. Multiple user clicks are converted to high-dimension queries, then lightweight decoder is used to segment multiple objects concurrently in 3D scene. Experimental results partly demonstrate the effectiveness of AGILE3D compared to previous models.", "questions": "Please see the weakness part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The interactive approach AGLIE3D can segment multiple objects simultaneously with limited user clicks. Compared to previous single-object iterative models, the proposed approach can reduce annotation time.\n2. Sufficient experiments have been conducted to show the promising results of the proposed method.", "weaknesses": "1. How does the model correct wrongly segmented object instance? For example, in Figure 6, if the initial segmentation wrongly groups two chairs into one instance, how can a later click fix this mistake?\n2. In Table 9, results 2 and 7 show minor performance drops without the C2C attention and temporal encoding components. Does this indicates that modeling the relations between clicks is less important for the proposed method?\n3. During training and testing, the clicks are sampled at the center of the object regions. Were there any ablation experiments done using other click sampling methods? How does the choice of click sampling influence performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u8W1FfFPEr", "reviewer": "Reviewer_Gxfu", "summary": "This paper introduces the novel interactive multi-object 3D segmentation task with a first multiple objects interactive segmentation approach, named AGILE3D.\nThe author also provides the setup, evaluation, and iterative training strategy for interactive multi-object segmentation on 3D scenes and conduct extensive experiments to validate the benefits of our task formulation.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The interactive multi-object 3D segmentation task is interesting and novel.\n2. The author provides a complete task process, which will be helpful for subsequent research.\n3. The proposed AGILE3D appears to be concise and effective, and is able to achieve state-of-the-art performance. At the same time, the author performed Efficiency Comparison, which is necessary for user-interactive tasks.", "weaknesses": "1. Is the contribution of this article incremental compared to 2D interactive segmentation?\nSince I am not an expert in this field, I cannot judge the extent of this article's contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)