Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Authors: Jie Liu
Pan Zhou
Zehao Xiao
Jiayi Shen
Wenzhe Yin
Jan-Jakob Sonke
Efstratios Gavves
Venue: ICML 2025 poster (2025)
Keywords: Interactive 3D Segmentation; Probabilistic Model; Neural Processes; Uncertainty Estimation
Primary area: applications->computer_vision
TL;DR: We propose the first probabilistic model for interactive 3D segmentation with novel hierarchical neural process, enabling effective few-shot generalization and providing reliable uncertainty estimation

Abstract: Interactive 3D segmentation has emerged as a promising solution for generating accurate object masks in complex 3D scenes by incorporating user-provided clicks. However, two critical challenges remain underexplored: (1) effectively generalizing from sparse user clicks to produce accurate segmentations and (2) quantifying predictive uncertainty to help users identify unreliable regions. In this work, we propose \emph{NPISeg3D}, a novel probabilistic framework that builds upon Neural Processes (NPs) to address these challenges. Specifically, NPISeg3D introduces a hierarchical latent variable structure with scene-specific and object-specific latent variables to enhance few-shot generalization by capturing both global context and object-specific characteristics. Additionally, we design a probabilistic prototype modulator that adaptively modulates click prototypes with object-specific latent variables, improving the model’s ability to capture object-aware context and quantify predictive uncertainty. Experiments on four  3D point cloud datasets demonstrate that NPISeg3D achieves superior segmentation performance with fewer clicks while providing reliable uncertainty estimations.

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Jie Liu Pan Zhou? 2
Zehao Xiao
Jiayi Shen
Wenzhe Yin
Jan-Jakob Sonke Efstratios Gavves'
Abstract
to target objects, while negative clicks define background areas. Through iterative user interaction, models can pro- gressively refine their segmentation predictions, ensuring adaptability and improved accuracy. This inherent flexibility has made interactive 3D segmentation a vital tool for a wide range of real-world applications including autonomous driv- ing (Ando et al., 2023), robotic manipulation (Zhuang et al., 2023), and augmented reality (Alhaija et al., 2017).
Interactive 3D segmentation has emerged as a promising solution for generating accurate ob- ject masks in complex 3D scenes by incorpo- rating user-provided clicks. However, two crit- ical challenges remain underexplored: (1) ef- fectively generalizing from sparse user clicks to produce accurate segmentation and (2) quanti- fying predictive uncertainty to help users iden- tify unreliable regions. In this work, we propose NDDSeg3D, a novel probabilistic framework that builds upon Neural Processes (NPs) to address these challenges. Specifically, NPISeg3D intro- duces a hierarchical latent variable structure with scene-specific and object-specific latent variables to enhance few-shot generalization by capturing both global context and object-specific charac- teristics. Additionally, we design probabilistic prototype modulator that adaptively modulates click prototypes with object-specific latent vari- ables, improving the model's ability to capture object-aware context and quantify predictive un- certainty. Experiments on four 3D point cloud datasets demonstrate that NPISeg3D achieves superior segmentation performance with fewer clicks while providing reliable uncertainty esti- mations. Project Page: https://jliuuai. github. Cechub..o/NPIIII
Recent advancements in the field have made substantial progress. For instance, single-object segmentation meth- ods for 3D point clouds (Kontogianni et al., 2023) and multi-object segmentation frameworks (Yue et al., 2023) have significantly pushed the boundaries of interactive 3D segmentation. However, two critical challenges remain in- sufficiently addressed, hindering the broader adoption and effectiveness of existing methods. The first one is effective few-shot generalization. Interactive 3D segmentation re- quires models to deliver accurate results with minimal user input. In real-world scenarios, users expect precise segmen- tation with only a few clicks, requiring models to generalize effectively from sparse user-provided cues. This challenge is amplified in diverse environments with complex scenes and wide variety of objects, where limited supervision must suffice for reliable segmentation (Schult et al., 2023; Takmaz et al., 2023; Kweon et al., 2024).
The second one is robust uncertainty estimation. The re- liability of interactive 3D segmentation models is highly sensitive to variability in user clicks, particularly when in- put is sparse or ambiguous (Zhou al., 2023). Uncertainty estimation is critical for interpreting model predictions, such as in model reliability assessment (Wu et al., 2023; Xu et al., 2023; Gong et al., 2023), identifying regions that require further user refinement, and guiding subsequent user inter- actions. Furthermore, reliable uncertainty quantification is essential in high-stakes applications, such as medical imag- ing (Rakic et al., 2024) and autonomous driving (Michel- more et al., 2020; Shafaei et al., 2018), where erroneous predictions can have significant consequences. Addressing these challenges is pivotal for advancing the capabilities of interactive 3D) segmentation and enabling its effective application across complex real-world scenarios.
1. Introduction
Interactive 3D segmentation (Kontogianni et al., 2023; Yue al., 2023; Valentin et al., 2015; Shen al., 2020; Zhi et al., 2022; Zhang al., 2024) seeks generate precise object masks in complex 3D environments by incorporating user-provided feedback, typically in the form of positive and negative clicks. Positive clicks indicate regions belonging
University of Amsterdam, Amsterdam, The Netherlands "Singapore Management University. Singapore Netherlands Can- cer Institute, Amsterdam, The Netherlands. Correspondence to: Jie Liu <j.liuS@uv.nl,, Pan Zhou <panzhou3@ gmail.com>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
Contributions. This work proposes NPISeg3D, novel probabilistic interactive 3D segmentation framework based

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
on hierarchical neural processes (NPs), to simultaneously address the challenges of few-shot generalization and uncer- tainty estimation. Our contributions are highlighted below.
(SoTAs) in both single- and multi-object interactive 3D seg- mentation tasks. For instance, on the KITTI-360 dataset, NPISeg3D achieves significant performance improvement of 8.4% and 4.2% over AGILE3D for single- and multi- object segmentation, respectively. Moreover, unlike exist- ing SoTA approaches such as InterObject3D and AGILE3D which neglect uncertainty estimation, NPISeg3D provides reliable and interpretable uncertainty qualification model predictions for interactive 3D segmentation.
Firstly, we introduce the first NP-based framework for in- teractive 3D segmentation, leveraging the strong few-shot generalization and uncertainty estimation capabilities of NPs (Jha al., 2022; Garnelo et al., 2018b) as shown in other domains like continual learning (Jha al., 2024) and semi-supervised learning (Wang et al., 2022; 2023). In our NP segmentation framework, user-provided clicks are treated as the context set providing partial observations about objects of interest, while the remaining unlabeled 3D points in the scene constitute the target set for predic- tion. Then, our NP framework formulates interactive 3D segmentation as a probabilistic modeling problem, where user-provided clicks (context set) are used to infer object- specific latent variables, and segmentation predictions are generated as predictive distribution over the target set. This probabilistic formulation enables the model to adaptively update predictions based on user-provided clicks while in- herently quantifying uncertainty in its predictions.
2. Related Work
Interactive 3D Segmentation. Despite its importance, in-
Shen et al., 2020; Zhi et al., 2022; Kontogianni et al., 2020; Yue et al., 2023; Zhang et al., 2024; Zhou et al., 2024) remains rela- tively underexplored. Pioneering works such as InterOb- ject3l> (Kontogianni et al., 2023) and CRSNet (Sun et al., 2023) have established click-based simulation schemes for interactive point cloud segmentation. AGILE3D (Yue et al., 2023) further advances the field by introducing an attention- based model to facilitate interactive multi-object segmenta- tion in 3D point clouds. Additionally, InterPCSeg (Zhang et al., 2024) integrates off-the-shelf semantic segmentation networks to improve their performance using corrective clicks at test-time. Unlike the aforementioned deterministic models, this work introduces the first probabilistic frame- work for interactive 3D segmentation, providing reliable and insightful uncertainty estimation to guide user interaction.
teractive 3D segmentation (Valentin et al., 2015;
Moreover, we develop hierarchical latent variable struc- ture to further improve the few-shot generalization of our NP segmentation framework. In complex scenes, NP mod- els with a single latent variable often struggle to capture the global structure and adequately model uncertainty (Guo et al., 2023). This challenge is particularly critical in multi- object interactive 3D segmentation setting ( (Yue et al., 2023) where understanding intricate scene layouts and onter- relationships is crucial for accurate segmentation. To solve this issue, we introduce hierarchical latent variable struc- latent variables.
Neural Processes. Neural Processes (NPs) (Garneo et al., 201 8b) learn approximate stochastic processes by mod- eling marginal distributions over functions. Conditional Neural Processes (CNPs) (Garneo et al., 2018a) extend this framework by learning predictive distributions conditioned on context and target sets. Various extensions have been proposed, such as Attentive Neural Processes (ANPs) (Kim et al., 2019), which leverage attention mechanisms for im- proved representation aggregation, and Transformer Neural Processes (TNPs) (Nguyen & Grover, 2022), which employ transformer architectures to model long-range dependencies. Further advancements focus on enhancing predictive accu- racy (Lee et al., 2020), stationarity (Foong et al., 2020), and robustness to noise (Kim et al., 2022). this work, we ex- tend NPs interactive 3D segmentation, enabling effective few-shot generalization and uncertainty estimation.
ture with scene-specific and The former models the global scene context and spatial re- lationships between objects, while the latter, derived from user-provided clicks, captures the fine-grained character- istics of each individual object. This hierarchical design enables the model to effectively integrate global scene un- derstanding with object-aware and click click-guided refinement, significantly enhancing its few-shot generalization.
Finally, we propose a probabilistic prototype modulator that dynamically integrates -bjeci--peccficc latent variables into each click prototype, which serves as a localized classi- fier to guide the segmentation process. By enriching these prototypes with object-aware context, the model derives more informative and adaptive click prototypes, thereby im- proving its ability to generalize from limited localized user clicks. Furthermore, the probabilistic nature of the modu- lator, achieved by latent space sampling, enables explicit modeling of prediction uncertainty, offering more reliable and interpretable insights into segmentation confidence.
3. Preliminary: Neural Processes
Neural Processes (NPs) (Garnelo et al., 2018a;b) are a family of methods designed to approximate the proba- bilistic distribution over continuous functions conditioned on partial observations. Formally, given context set Dc (Xc, Yc) and target set Dr (Xr, Yr), where
Extensive experiments on four benchmark datasets demon- strate the superiority of NPISeg3D over state-of-the-arts
2

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Hierarchical Latent Variables
Probabilistc Prototype Modulator
Training only Sampling
(Xe, Yc)
x
Point Encoder
Ap (2g)
p(z)
MLPs
(v.P)
Click set C
Click prototypes
(Context)
Scene-level
Aggregator
Prior
Object-level
Agg Aggregator
X
Superivison:
Lxeg (Pr. Y+)+ !
Posterior
(Target)
Scale & Shift
Mask Head
Scene features
((3))
(2,)
8
Target points X7
3D scene
Mask prediction p(Fr)
Figure 1. Overview of NNPSSggD, We formulate interactive 3D segmentation probabilistic modeling problem with neural processes. Given a 3D scene S and user click set C, point encoder encodes them into click prototypes X (context data) and scene fcatures Xr (target data). Then, we introduce two hicrarchical latent variables: scene-level latent variable x, and object-level latent variable zo, to enable probabilistic modeling and capture contextual information across hierarchical levels. In probabilistic prototype modulator, cach object-specific latent variable utilized to generate thereby enhancing few-shot generalization and providing reliable uncertainty estimation. The posterior distributions of the latent variables are inferred from the target set (Xr, Yr). which supervise the prior during training.
weights (r. which modulate its corresponding click prototypes.
X and Y represent onput-output pairs, NPs aim to model the conditional distribution
4. Methodology
Xr,
Task Definition. Interactive multi-object 3D segmentation operates 3D scene represented as point cloud, denoted by S E N2x where N is the number of 3D points. Each point is characterized by its spatial coordinates (z, 2) and color features (r,g. b). Then, users provide sequence of in- teraction clicks = {(Ck, ox) ow)]{= where C₁ = (Zx, yk 24) denotes the 3D coordinates of the k-thh click, og E (1) is the click label, with = 0 representing the background and I . .M denoting the remaining M (> 1) ob- jects. Given the 3D scene S and click sequence C, the goal is to predict the segmentation mask Y E R.,, where each element Y, E (1)/ denotes the label of the i-th point in the scene. Users can iteratively provide corrective clicks to refine segmentation results until they are satisfactory.
Generally, NPs model the distribution over functions by introducing a global latent variable z, which cap- tures the underlying uncertainty in the function space. This latent variable is sampled from prior distribution p(z) (D Jeonditioned on the context set De. Given z, the model infers the target ouputs Yr based on the conditional distribution p oY [x,, z). where Xx denotes the target in- puts. Thus, the overall NPs model is formulated as:
P(Yr|XT Dc) DDD
2)p(z(Dc)d.
Due to the intractable true posterior, previous work resort amortized variational inference (Kingma & Welling, 2013) to optimize the NPs model. Specifically, let (()(Dr) repre- sent the variational posterior of the latent variable z inferred from the target set Dr during training, the evidence lower bound (ELBO) for NPs derived as:
4.1. NP Framework for Interactive Segmentation
To enable effective few-shot generalization for interactive 3D segmentation with robust uncertainty estimation, we for- mulate the problem within Neural Processes (NPs) frame- work, a probabilistic approach. The NPs framework pro- vides a natural formulation for interactive 3D segmentation, where asoviddovided clicks serve as the context set, and the unclicked points constitute the target set to be predicted. We define the context set and target set in the feature space. As shown in Figure 1, given the 3D scene and input clicks set C, a point encoder (Kontogianni al., 2023; Yue et al., 2023) generates corresponding features X7 E R for the 3D M
log (Ygp(Y Dc) 2 En(prr [log p( Y 2)]
DKL. ((l(a (2)
The first term of the ELBO is the predictive log-likelihood, which encourages the model generate accurate predictions for the target outputs Yr. The second term is a Kullback- Leibler (KL) divergence that regularizes the variational pos- terior (z) to stay close to the prior p(z(Dc). This probabilistic formulation enables the model to generalize from few observed samples (context) to make predictions on unseen data (target), inherently supporting few-shot gen- eralization and uncertainty estimation (Jakkala, 2021: Wang & Van Hoof, 2022; Shen et al., 2023; Wang al., 2025).
points and X (X Im for the clicked points. For ob- m, No
ject m, X = C Ji=1 is the features of N clicks with each click feature X E Rdd and Y = (Y Ji=i are corresponding one-hot labels. We refer X as "elick pro- totypes" for object m, they serve click-wie classifiers for segmentation (Yue et al., 2023). Then, the context set is
No
3

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
(Xc, xc)=((X Y while the target set is = D== Yr). Given the context and target set, NP aims to model the prediction distribution p( (T, Dc).
DC
Zz
Context set
Latent variables
To effectively model the diverse and complex structures of multiple objects within a 3D scene, we assume that the segmentation functions for each object m are conditionally independent given their respective context information. In interactive segmentation, user clicks are typically provided for distinct objects, which aligns with our assumption of processing each object independently. Specifically, for each object, we introduce an object-speciftt latent variable z," E Rd to capture fine-grained object-specific characteristics. The latent variable z is conditioned on the context set De = ( Yd) for object m. Considering that click prototypes X inherently encapsulates both feature and label information of clicks, we represent the context set De as X for simplicity in the following equations. Then, the segmentation function for object m is defined by the following conditional distribution:
D
0
Z
Zo
YT
Target points
: D
D ...
D ... D
X
Target labels
Figure 2. Computational Graph of our NPISeg3D. The frame- work incorporates hierarchical inference structure with scene- level latent variable (2a) and an object-level latent variable (2o). capturing contextual information at different spatial levels.
4.2. Hierarchical NP Framework for segmentation
To enhance few-shot generalization in complex scenes, we introduce a hierarchical latent variable structure into the NP framework in Eq. (4). As illustrated Figure 2, this hierarchal structure consists of scene-specific latent vari- able 2, E Rdd and M 1 object-specific latent variable z Generally, the scene-specifit latent variable z, is de- signed to capture the global scene context and inter-object re relationships, providing a holistic understanding of the 3D scene. Simultancously, the object-specific latent variables 2," focus on encoding fine-grained and object-specific de- tails specified by user- ser-ppovided clicks. By incorporating the hierarchal latent variables structure, the NP framework in Eq. (4) is reformulated as:
P( Xx, Xr, D8)= /XYFAX 23 p(Z"
X
dz," C
(3) where p(z," X denotes the prior distribution of the latent variable z inferred from the context click prototypes X p( (YFl z²") is the predictive distribution conditioned on the latent variable z², modeling the probability of each point in Xy belonging to object m. By considering multiple objects, the joint conditional distribution is formulated as:
M p Xr, Dc) = II
P(
Xr. X0,2" 20, 2.)
M pl XY Dc)= = II
171 m0
p(
Th T
XT.
200 p(z X m
dz
p( Th 12., x dzz"
plza Xc) dz.,
(5)
(4)
In practice, the prior distribution p(z Th X is parameter- ized by a small learnable network. During training, the prior is optimized to approximate the posterior inferred from target data, as described in Eq. (2). This enables the NP framework to effectively capture object-specific char- acteristics from limited number of user- speriird clicks, facilitating rapid adaptation to unseen objects and enhancing the model's few-shot generalization capability in interac- tive segmentation. Moreover, the probabilistic formulation in Eq. (4) inherently enables the model to provide uncer- tainty estimation for segmentation results, offering valuable feedback for guiding user interactions.
where p(z,, Xc) denotes the prior distribution of the scene- specific latent variable, encoding holistic scene context from all user clicks. p(z." (z., X models the object-specific latent variable conditioned on the scene-specific latent vari- able 2, and click prototypes X of object m. Lastly, p( YT [ X z. 2.) defines the predictive distribution over the target set for object m, and here we introduce an additional condition, i.c., the click prototypes X incor- porate click-level details.
This hierarchical design allows the model to seamlessly integrate fine-grained object-specific guidance with global scene understanding, effectively enhancing few-shot gener- alization and leading to more accurate segmentation. Next, we describe the inference process of these latent variables.
Despite its effectiveness, modeling the distribution of func- tions with only object-level latent variables potentially limits the few-shot generalization capacity of the model (Guo et al., 2023; Shen et al., 2021). This limitation becomes more pronounced in complex scenes, particularly multi-object in- teractive segmentation settings, where understanding global scene context and inter-object relationships is crucial. To address this challenge, we propose the solution in Sec. 4.2.
Scene-Specific Latent Variable. As illustrated in Figure 1, we infer the distribution of the scene-specific latent variable via scene-level aggregator. Specifically, we first compute object-level prototypes X by averaging N click pro- totypes {X (x No Je for each object m. Then, object-level

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
prototypes {X mm are averaged to construct scene- level prototype Xc. To capture interactions between the scene- and object-level prototypes, we employ lightweight single-layered transformer (Trans), and the scene-specific latent variable is inferred as follows:
into the model to represent uncertainty. ((2"")) E Rd and B(25).)) E Rd are the scale and shift parameters, respec- tively, computed via a two-layer multi-Layer perceptron (MLP) conditioned on zm-.. @ denotes element-wis multi- plication, enabling feature-wise modulation. X denotes the modulated click prototype for object m derived from the i-th user-provided click and the j-th Monte Carlo sample.
[Ma 0a] = MLP (Trans (Xo:X I))),
(6)
where [s] are the mean and variance of the Gaussian distribution p(z, x)) generated using a two-layer multi- layer perceptron (MLP). The scene-specific latent variable z captures the global scene context, which is crucial in chal- lenging scenarios with overlapping objects or ambiguous boundaries, where localized guidance alone is insufficient.
By coupling latent variables with click prototype modula- tion, the framework creates seamless information flow: Scene - Objects + Clicks. This hierarchical struc- ture captures multi- multi-granularityy context, including global scene layout, object-specific characteristics, and click-level details, significantly improving few-shot generalization and leading to more accurate segmentation. Moreover, by draw- ing multiple samples z from the prior distribution, we enable the prototype modulator to be probabilistic, allowing the model to estimate uncertainty in its predictions.
Object-Specific Latent Variables. For each object m, the object-specific latent variable zn" is conditioned on the scene-specific latent variable Zg and the correspond- ing object-specific click prototypes X This allows z to integrate global scene context while adapting to fine- ane-grained and click-specific details. Formally, 2 is generated through an object-level aggregator, illustrated in Figure 1. This process is formulated as follows:
4.4. NPISeg3D Pipeline
Evidence Lower Bound. To optimize NPISeg3D in Eq. (5), we adopt variational inference (Kingma & Welling, 2013) and derive the evidence lower bound (ELBO) as:
[p m o m
MLP oZ. + (1- Σ x
log p( Y Xx, Dc) 2
M Bo log Y Xr, X 2," 2.)
where z, is Monte Carlo sample drawn from the prior dis- tribution p(z,, Xx)), and a E [0. 1] balances the scene-level context and the object-level context introduced by clicks. The object-specific latent variables z effectively capture object-aware context and model object-level uncertainty, enabling robust few-shot generalization.
Ee(2, Xx)
m=0
DKL. [g(2)" Th Ze, X )
I|p(2 12.0 Xx:]
DxL [9(2, X2 [p(z,
where q0 Xr) and as (0 z. X denote the varia- tional posteriors of latent variables z, and z." and parame- terized by and øi respectively. The variational posteriors are inferred from the target set (Xr, Yr), and X denotes all target point features belonging to object m. The prior distributions are supervised by the variational posterior us- ing Kullback-Leibllr (KI.) divergence, guiding the model to effectively capture richer object-specific information with limited context data and enabling better generalization to unseen target data. See derivations Appendix A.
4.3. Probabilistic Prototype Modulator
Click prototypes X encapsulate fine-grained and localized cues for segmenting object m and collectively define its de- cision boundary as an ensemble classifier (Yue et al., 2023). However, each individual click prototype X often lacks awareness of the broader object-level or scene-level context and struggles to effectively capture predictive uncertainty.
To this end, we introduce a probabilistic prototype mod- ulator that dynamically adjusts click prototypes using the corresponding object-specific latent variables z." These latent variables, derived from our hierarchical modeling ap- proach, capture spjecc-specific uncertainty while integrating global scene context, effectively enriching click prototypes with high-level semantic information. Specifically, the mod- ulated click prototype X
Model Training. The loss function for NPISeg3D is derived from the ELBO in Eq. (9). In practice, the first term is implemented segmentation loss that enforces alignment between predictions and ground truth. The second term consists of two KL divergence regularization terms that constrain the latent variable distributions. Therefore, the overall training objective is formulated as:
(kd. for object m is defined as:
X (2 @ x + B)
L = C=Coog YT) + dxx (PKL (r)zx
where X E Rdd is the i-th click prototype for object m, znd E Rd is the j-th Monte Carlo sample drawn from the prior distribution p(z"") l,,, X introducing stochasticity
DKL (zzz"Zzz, X+) z. xx)) (10)
5

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Methods
Train Eval
IoU@5 IoL'@00
IoU@
Avg
NoC@80
NoC@85 1 NoC@90
Avg
InterObject3D
75.1
80.3
81.6
79.0
10.2
13.5
16.6
13.4
InterObjectSD++
ScanNet40- - ScannNe140
79.2
82.6
83.3
81.7
8.6
12.4
15.7
12.2
AGILESD
(In-domain)
82.3
85.0
86.0
84.4
6.3
10.0
14.4
10.2
NPISeg3D (Ours)
82.6
85.2
86.2
84.7
10.0
14.4
10.1
InterObject3D
76.9
85.0
87.3
83.1
6.8
8.8
13.5
9.7
InterOheetttD++
ScanNetd0=S3DDIS-o
81.9
88.3
89.3
86.5
7.6
11.6
8.3
AGILE3D
(Out-of-domain)
86.3
88.3
90.3
88.3
3.4
5.7
9.6
6.2
NPISeg3D (Ours)
89.0
90.9
91.5
90.5
7.8
InterObject3D
73.2
83.7
87.0
81.3
7.7
10.8
18.4
12.3
InterObjett3D++
ScunNer40-RRppie
80.4
87.5
88.8
85.6
7.5
15.7
9.7
AGILE3D
(Out-of-domain)
83.5
87.9
89.5
86.9
3.6
5.8
14.1
7.8
NPISeg3D (Ours)
85.7
89.3
90.4
88.5
4.7
13.0
InterObject3D
10.5
22.1
31.0
21.2
19.8
19.8
19.9
19.8
InterObject3De+
ScanNer00=KKITTI-36
16.7
52.2
35.3
18.9
19.3
AGILE3D
(Out-of-domain)
40.5
44.3
48.2
44.3
17.4
18.3
18.8
18.2
NPISeg3D (Ours)
44.0
48.5
52.9
48.5
16.4
17.0
17.6
17.0
Table 1.
Quantitative results on interactive multi-object segmentation. Avg denotes the mean IoU across 5, 10, and 15 clicks, or the mean NoC across 80%, 85%, and 90% IoU thresholds. NPISeg3D a consistent lop-performer, particularly on out-of-domain datasets.
where Leeg (Y Yr) is the segmentation loss implemented as the combination of dice loss and cross-entropy loss (Yue
Scene & GT
et al., 2023; Schult et al., 2023), Y denotes the predicted mask for the target point cloud Xr, and Agı is a balancing coefficient for the regularization terms.
AGILE3D
Model Inference. As shown in Figure I, given the mod- ulated click prototypes and target points, we generate the final segmentation mask via mask head. Specifically, we first compute the segmentation logits YI for object m by aggregating predictions across multiple Monte Carlo sam- ples. This captures the uncertainty introduced by the prob- abilistic latent variables. Then, per-point max operation is applied to select the most confident response across all eser-provided clicks. This process is formulated as:
IoU@6666.7
IoL 101-119
loU@5=18866
ToU@@ 10=20.6
NPISeg
ToU:@6-77.8
1oU@88-4.0
1oU@55-42.4
10-110-52.7
S3DIS
KITT360
Figure 3. Qualitative results on interactive multi-object segmen- tation S3DIS and KITTI360. Newly added clicks repre- sented hy dark-colored dots. Please zoom in for more details.
Nio cos cos(Xr x.r, X...).),,
Ym
max NEI Now N
(11)
and KITTI-360 (Liao al., 2022), outdoor LiDAR dataset designed for autonomous driving scenarios. Further details on the dataset setup are provided in Appendix D.1.
where cos(..) represents the cosine similarity, and N₂, is the number of Monte Carlo samples used to approximate the latent distribution. After computing the logits {Ý m for M objects (including background), the final segmen- tation mask Yr by assigning each point the object with the highest similarity score. We further obtain the uncer- tainty map by replacing the mean aggregation in Eq. (11) with their variances, see details in Appendix B.5.
Evaluation Metrics. We follow prior works (Kontogianni et al., 2023; Yue et al., 2023; Zhang et al., 2024) and evaluate model performance using two key metrics: (1) NoC@q % I. which measures the average number of clicks required to reach target IoUs of 80%, 85%, and 90%, denoted as NoC@ 80, NoC @ 85, and NoC@90, (2) IoU @ k 1. which evaluates the average IoU achieved after 5, 10, and 15 clicks, represented as lo loU@ 10, and @ 15. A maximum of 20 clicks is allowed per instance, and results are averaged across all instances in the multi-object segmentation setting.
5. Experiments
Datasets. We follow the dataset setup in prior work (Yue al., 2023) and train our model the ScanNetV2-Tras dataset (Dai et al., 2017). For evaluation,
Baselines. We compare NPISeg3D with three interactive 3D segmentation methods: InterObject3D (Kontogianni et al., 2023), InterObject3D+++ (Yue et al., 2023), and AGILE3) (Yue et al., 2023). InterObject3D processes objects sequen- tially multi-object scenarios, while InterObject3D+ en- hances its performance in multi-object settings with iterative training. AGILE3D adopts an attention sitentiio-based framework,
consider two types of datasets: (1) In-domain dataset: ScanNetV2- Val (Dai et al., 2017), which shares the same distribu- tion as the training data; (2) Out-of-domain datasets: S3DIS (Armeni et al., 2016), collected using different sen- sors; Replica (Straub al., 2019), synthetic indoor dataset;
6

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Methods
Train Eval
Evall
loU@5
IoU@10
IoU@
Avg
NoC@80 & NoC@85 & NoC@90
Avg
InterObject3D
72.4
79.9
82.4
78.2
8.9
11.2
14.2
11.4
InterOheettDD++
78.0
82.9
84.2
81.7
10.0
13.2
10.3
AGILE3D
(In-domain)
79.9
83.7
85.0
82.9
7.1
9.6
12.9
9.9
NPISeg3D (Ours)
80,5
83.8
85.0
83.1
9.5
12.8
InterObject3D
72.4
83.6
88.3
81.4
6.8
8.4
11.0
8.7
InterObject3D4+
80.8
89.2
91.5
87.2
6.7
9.3
7.1
AGILE3D
(Out-of-domain)
83.5
88.2
89.5
87.1
4.8
6.4
9.5
6,9
NPISeg3D (Ours)
85.3
89.2
90.1
88.2
6.0
8.9
6.4
InterObject3D
64.4
80.1
86.1
76.9
8.4
10.0
12.4
10.3
InterObject3D++
ScanNet00-RRppie
72.6
83.8
86.7
81.0
8.1
11.2
AGILE3D
(Out-of-domain)
76.3
85.6
87.1
83.0
5.7
7.9
10.6
8.1
NPISeg3D (Ours)
78.9
87.4
88.7
85.0
6.3
9.7
InterObject3D InterOhject3D++
14.3
26.3
35.0
25.2
19.1
19.4
19.7
19.4
ScanNet40 -67
19.9
40.6
55.1
38.5
17.0
17.7
18.4
AGILESD
(Out-of-domain)
44.4
49.6
54.9
49.6
14.2
15.5
16.8
15.5
NPISeg3D (Ours)
55.7
$7.5
60.9
$8.0
11.5
13.0
14.7
13.1
Table 2. Quantitative results on interactive single-object segmentation. Avg denotes the mean IoU across 5, 10, and 15 clicks, or the mean NoC across 80%, 85%, and 90% ToU thresholds. NPISeg3D a consistent top-performer, particularly on out-of-domain datasets.
Methods
Datasets
loU@1
IoU@21
IoU@3
S3DIS
KITTI360
Z, Zo Mp
InterObject3D
38.5
54.0
62.5
mloUG @ 10
NoC @58)
mloU@ @ 101
NoC@85,
InterObject3D++
S3DIS
32.7
69.0
AGILE3D
58.5
70.7
77.4
88.7
5.5
18.1
NPISeg3D
60.5
74.0
80.0
90.3
5.0
46.1
17.7
4
90.7
4.7
46.7
17.4
InterObject3D
34.7
46.7
56.7
InterObject3D++
Replica
21.5
41.5
55.1
90.4
4.7
47.8
17.2
AGILE3D
53.4
63.7
67.4
90.9
4.4
48.5
17.0
NPISeg3D
54.6
65.7
70.3
Table 4. Ablation study of model components on multi-object segmentation task. Ze and Zo denote the seene-specific and object-specific latent variables, respectively, while Mp represents probabilistic prototype modulator.
InterObject3D
2.0
5.1
8.5
InterOhjecttDD++ KITTI-360
3.4
7.0
11.0
AGILE3D
34.8
40.7
42.7
NPISeg3D
36,9
46.5
52.0
Table Quantitative results on single-object segmentation task with limited clicks (S 3). The best results are highlighted in bold.
strate the superiority of NPISeg3D enhancing few-shot generalization out-of-domain datasets, attributed to probabilistic framework with hierarchical NPs.
achieving state-of-the-art (SoTA) performance.
Qualitative results. Figure 3 presents the qualitative com- parison between AGILE3D and NPISeg3D under the same number of clicks. NPISeg3D predicts more accurate seg- mentation masks, especially on the challenging KITTI-360 dataset, where it achieves 42.4% IoU@5, significantly out- performing AGILE3D's 18.6% IoU@5. These results derscore NPISeg3D's capability to achieve high-quality seg- mentation with a limited number of user clicks.
5.1. Evaluation Multi-object Segmentation
In-domain Results. As shown in Table NPISeg3D per- forms competitive and even better than baseline methods on the ScanNet40 dataset by achieving higher segmentation accuracy with fewer user interactions. For example, it at- tains an loU@5 82.6 and reduces the number of clicks required achieve 80% IoU (NoC@S) 80) 5.9, compared 6.3 for AGILE3D. These results highlight the superiority of NDISeg3D in achieving high-quality interactive segmen- tation on data distributions similar to the training set.
5.2. Evaluation on Single-object Segmentation
Results. As shown in Table 2, NPISeg3D achieves com- petitive or superior performance compared to AGILE3D on the ScanNet dataset, attaining an loU@ 5 of 80.9% versus AGILE3D's 79.9%. Moreover, NPISeg3D again outper- forms most baselines on the out-of-domain datasets. For instance, on KITTI-360, which features complex outdoor LiDAR scenes with substantial domain shifts, NPISeg3D achieves an IoU of 55.7% with 5 clicks, markedly surpass- ing AGILE3D's 44.4%. These results highlight the superior generalization capability of NPISeg3D in the single-object segmentation setting. See more results Appendix B.7.
Generalization to Out-of-domain Notably, NPISeg3D demonstrates strong few-shot generalization capabilities on out-of-domain datasets, including S3DIS-A5, Replica, and KITTI-360, achieving superior segmentation performance with fewer user interactions. As shown in Table (In. 2), on the -3 dataset, NPISeg3D achieves an impres- sive 89.0% loU @5 (+2.7% over AGILE3D) while reduc- ing NoC 80 to 2.8 compared to 3.4 for AGILE3D, Simi- larly, on the challenging KITTI-360 dataset, attains 52.9% loU@5 (+4.7% over AGILE3D) and lowers NoC@90 to 17.6, outperforming AGILE3D's 18.8. These results demon-
Low-click Results, Our method demonstrates remarkable

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
S3DIS
KITTI360
User
Data Source
IoU@3 f
Io[@3 f
NoC@80
t@s0
Modulation
mloU@ 10f NoC@ 85| m 10 NoC@85
Simulator
S3DIS
84.8
92.7
2.2
Human
83.2 1 0.6
93.5 + 0.7
1.7=0.3
3 min
Concat
90.6
4.7
47.8
17.4
Simulator
79.3
86.6
3.5
KITTI-360
Add
90.4
4.9
48.2
17.2
Human
81.1 = 1.4 88.1 +: 0.4
2.8 = 0.4
5 min
Ours
89.6
5.0
46.1
17.8
Table 7. User study on annotating 20 objects. i@80 represents the time required reach 80% IoU.
Ours
90.9
4.4
48.5
17.0
Table Ablation study of prototype modulation strategies. and & denote probabilistic and deterministic modulations.
S3DIS
KITTI360
Method
mlot @ 10
NoC@85, mlot @ 10 NoC@85,
Ground Truth
ToU@2-44.5
1oU@4-74.6
loU@6-81.6
MC dropout (r 0.1)
90.1
5.0
40.6
18.2
MC dropout (r 0.3)
90.4
4.8
38.8
18.9
MC dropout (r = 0.5)
88.9
6.3
24.9
19.3
NPISeg3D (n 5)
90.8
4.5
48.6
17.0
NPISeg3D (n 10)
90.9
4.4
48.5
17.0
NPISeg3D 20)
90.8
4.5
48.5
17.0
3D scene
uncertainty@22 uncertainty@@4 uncertainty@@6
Table Ablation study of different uncertainty estimation methods with different parameters. r denotes the dropout rate, and n is the sampling number of latent variables in NPs.
Figure 4. Uncertainty maps of predictions with increasing clicks. We show mask predictions and uncertainty after K clicks.
explore alternative modulation strategies, such as Concat and Add, to assess different ways of integrating object- specific latent variables with click prototypes. The results validate the superiority of our method in achieving accurate segmentation with improved generalization.
performance in the low-click regime (S clicks) and out- of-domain scenarios. As shown in Table 3, with only two clicks, NPISeg3D attains IoU scores of 74.0% on S3DIS and 65.7% on KITTI-360. When increased to three clicks, the loU further improves to 80.0% on S3DIS and 52.0% on KITTI-360, significantly outperforming the state-of-the-an AGILE3D. These results furhter highlight superior few-shot generalization of our NPISeg3D with limited clicks.
Effect of Uncertainty Estimation Strategies. Table 6 compares MC Dropout (Xiang et al., 2022; Wang al., 2023) with varying dropout rates (r = 0.1, 0.3, 0.5) and NPISeg3D with different latent variable sampling numbers (n = 5, 10, 20). Generally, MC Dropout shows a decline in performance as the dropout rate increases, indicating reduced reliability in uncertainty estimation with higher dropout. In contrast, NPISeg3D maintains stable and supe- rior performance across different sampling numbers, demon- strating its robustness and more effective uncertainty mod- eling. This highlights the superior uncertainty modeling of our NPISeg3D with hierarchical latent variables.
5.3. Ablation and Analysis
Effect of Hierarchical Latent Variables. We conduct an ablation study on the S3DIS and KITTI-360 datasets to assess the impact of the hierarchical latent variables in NPISeg3D. As shown in Table 4, both the ceene-specific latent variable n, and the object-specific latent variable z, contribute to improved segmentation performance. For in- stance, on S3DIS, introducing z, reduces NoC@85 85 from 5.0 4.4 (In. 2 vs. ln. 6), highlighting its effectiveness in captur- ing object-specific context. With both x, and 2, (In. 6), our ND achieves the best results, validating the advan- tage of the hierarchical modeling in capturing multi-granular context and improving few-shot generalization.
User Study. We conducted a user study with real hu- man clicks evaluate NPISeg3D in practical scenarios. As shown in Table 7, real users achieved performance compa- rable the simulator, demonstrating the reliability of our method. Notably, our method consistently achieves superior and robust results across different user behaviors. This im- provement is attributed to its uncertainty estimation, which effectively guides user interactions by identifying uncertain regions, reducing the time spent searching for high-error regions. See Appendix C for further details.
Effect of Probabilistic Prototype Modulator. As shown Table 4 (In. 5 vs. In. 6), incorporating probabilistic proto- type modulator yields notable performance improvements, demonstrating its beneficial effect in improving segmenta- tion accuracy. Furthermore, Table 5 (In. 3 vs. In. 4) presents comparison with deterministic modulator, where the levect-level prototype X is directly used to generate mod- ulator parameters. The probabilistic approach offers sub- stantial performance gains by capturing uncertainty and enabling more adaptive responses to user inputs. We further
Qualitative Analysis of Uncertainty Estimation. As shown in Figure 4, NPISeg3D generates reliable and infor- mative uncertainty estimations. Notably, high uncertainty is primarily concentrated in incorrectly segmented regions, along the edges of the predicted mask, and in areas distant from user clicks, reflecting inherently challenging cases in
8

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
interactive 3D segmentation. Moreover, uncertainty progres- sively decreases as more user clicks are provided, indicating the model's increasing confidence in segmentation. For additional examples, see Appendix B.5 and Figure 7 and 8.
large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1534-1543, 2016.
Choy, C.,, Gwak, J., and Savarese, S. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3075-3084, 2019.
6. Conclusion
We present NPISeg3D. the first probabilistic framework for interactive 3D segmentation. NPISeg3D formulates the task within the NP framework, incorporating hierarchical latent variable structure and a probabilistic prototype modulator to enhance the model's capability in few-shot generalization and uncertainty estimation. Experiments demonstrate that NDISeg3D achieves high-quality segmentation and provides reliable uncertainty estimation across diverse 3D scenarios.
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M. Scannet: Richly-annotated 3d recon- structions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839) 2017.
Foong, A., Bruinsma, W., Gordon, J., Dubois, Y., Requeima, J., and Turner, R. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33: 8284 8295, 2020.
Limitations. While NDISeg3D demonstrates strong effec- tiveness, its out-of-domain performance still lags behind in-domain scenarios. Future work could address this by ex- panding the training set or incorporating domain adaptation techniques to further improve generalization.
Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T. Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D., and Eslami, S. A. Conditional neural processes. In Interna- tional conference machine learning, pp. 1704-1713, PMLR, 2018a.
Acknowledgments
This work was partially funded by Elekta Oncology Systems AB and RVO public-private Partnership grant (PPS2102). Pan Zhou was supported by the Singapore Ministry of Ed- ucation (MOH) Academic Research Fund (AcRF) Tier grants (project ID: ---SI--MMU-070..
Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. Neural pro- cesses. arXiv preprint 1707v:8007.1661 2018b.
Gong, J., Foo, L. G., Fan, Z., Ke, Q.. Rahmani. H., and Liu, J. Difffpose Toward more reliable 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13041-13051, 2023.
Impact Statement
This work contributes the field of machine learning by formulating interactive 3D segmentation within a proba- bilistic framework, enhancing few-shot generalization and uncertainty estimation. Our work has potential applications in areas such as medical imaging, autonomous driving, and robotics. While we do not foresee immediate societal risks, careful deployment is required in high-stakes applications.
Guo, Z., Lan, C., Zhang, Z., Lu, Y., and Chen, Z. Versatile neural processes for learning implicit neural representa- tions. arXiv preprint 20V.:2301.888.3 2023.
Jakkala, K. Deep gaussian processes: A survey. arXiv preprint arXiv:21066 12135, 2021.
References
Jha, S., Gong, D., Wang. X., Turner, R. E., and Yao, L. The neural process family: Survey, applications and perspec- tives. arXiv preprint arXiv:2209.005/ 00510 2022.
Alhaija, H. A., Mustikovela, S. K., Mescheder, L., Geiger, A., and Rother, C. Augmented reality meets deep learning for car instance segmentation in urban scenes. In British machine vision conference, volume 1, 2017.
Jha, S., Gong, D., Zhao, H., and Yao, L. Npel: Neural processes for uncertainty- aware continual learning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Ando, A., Gidaris, S., Bursuc, A., Puy, G., Boulch, A., and Marlet, R. Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5240-520555 52500, 2023.
Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., and Teh, Y. W. Attentive neural processes. arXiv preprint 1rXiv:1901.05700 2019.
Kim, M., Go, K., and Yun, S.-Y. Neural processes with stochastic attention: Paying more attention to the context dataset. arXiv preprint arXiv:2204.0544 2022.
Armeni, I., Sener, O., Zamir, A. R., Jiang, H., Brilakis, I., Fischer, M., and Savarese, S. 3d semantic parsing of

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint 131.vv13/2.66 20133
Shafaei, S., Kugele, S., Osman, M. H., and Knoll, A. Un- certainty in machine learning: A safety perspective on autonomous driving. In Computer Safety, Reliability, and Security: SAFECOMP 2018 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Västerås, Swe- den, September 18, 2018, Proceedings 37, pp. 458-464. Springer, 2018.
2013.
Kontogianni, T., Gygli, M., Uijlings, J., and Ferrari, V. Con- tinuous adaptation for interactive object segmentation by learning from corrections. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pp. 579-596. Springer, 2020.
Shen, J., Zhen, X., Worring, M., and Shao, L. Multi-task neural processes. arXiv preprint arXiv:2111.05820 2021.
Kontogianni, T., Celikkan, E.,, Tang. S., and Schindler, K. Interactive object segmentation in 3d point clouds. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2891-2897, IEEE, 2023.
Shen, J., Zhen, X., Wang. Q., and Worring, M. Episodic multi-task learning with heterogeneous neural processes. Advances in Neural Information Processing Systems, 36: 75214-75228, 2023.
Kweon, H., Kim, J., and Yoon, K.-. Weakly supervised point cloud semantic segmentation via artificial oracle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3721-3731, June 2024.
Shen, T., Gao, J., Kar, A., and Fidler, S. Interactive an- notation of 3d object geometry using 2d scribbles. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pp. 751-767. Springer, 2020.
Lee, J., Lee, Y., Kim, J., Yang, E., Hwang, S. J., and Teh, Y. W. Bootstrapping neural processes. Advances in neural information processing systems, 33:6606-6615. 2020.
Sofiiuk, K., Petrov, I. A., and Konushin, A. Reviving itera- tive training with mask guidance for interactive segmen- tation. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 3141 3141-3145. IEEE, 2022.
Liao, Y., Xie, J., and Geiger, A. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3) -310, 2022.
Straub, J., Whelan, T., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J. J., Mur-Artal, R., Ren, C., Verma, S., et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint
Michelmore, R., Wicker, M., Laurenti, L., Cardelli, L., Gal, Y.. and Kwiatkowska, M. Uncertainty quantification with statistical guarantees in end-to-end autonomous driv- ing control. In 2020 IEEE international conference robotics and automation (ICRA), pp. 7344-7350. IEEE, 2020.
2019.
Sun, W., Luo, Z., Chen, Y., Li, H., Marcato, J., Gonçalves, W. N., and Li, J. A click-based interactive segmenta- tion network for point clouds. IEEE Transactions on Geoscience and Remote Sensing, 2023.
Mo, K., Zhu, S., Chang, A. X., Yi, L.., Tripathi, S., Guibas, L. J., and Su, H. Partnet: A large-scale benchmark for fine-grate -raiira and hierarchical pert-level 3d object under- standing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 909-918, 2019.
Takmaz, A., Fedele, E., Sumner, R. W., Pollefeys, M., Tombari, F., and Engelmann, F. Openmask3d: Open- vocabulary 3d instance segmentation. arXiv preprint arXiv:2306.1 13631, 2023.
Valentin, J., Vineet, V., Cheng, M.-M., Kim, D., Shotton, J., Kohli, P.. Nießner, M., Criminisi, A., Izadi, S., and Torr, P. Semanticpaint: Interactive 3d labeling and learning at your fingertips. ACM Transactions on Graphics (TOG), 34(5):1-17, 2015.
Nguyen, T. and Grover, A. Transformer neural processes: Uncertainty-awart meta learning via sequence modeling. arXiv preprint ar (tt.:2007.04177 2022.
Rakic, M., Wong, H. E., Ortiz, J. J. G., Cimini, B. A., Guttag, V., and Dalca, A. V. Tyche: Stochastic in- context learning for medical image segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11159-1117,, 2024.
Wang, J., Lukasiewicz. T., Massiceti, D., Hu, X., Pavlovic, V., and Neophytou, A. Np-match: When neural processes meet supi-super learning. In International Confer- ence on Machine Learning, pp. 2291922222000 22934. PMLR, 2022.
Schult, J.. Engelmann, F., Hermans, A., Litany, O., Tang, S., and Leibe, B. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 8216-8223. IEFE, 2023.
Wang, J., Massiceti, D., Hu, X., Pavlovic, V., and Lukasiewicz, T. Np-semiseg: when neural processes meet semi-supervised semantic segmentation. In Inter- national Conference on Machine Learning, pp. 36138 36156. PMLR, 2023.
10

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Wang, Q. and Van Hoof, H. Learning expressive meta- representations with mixture of expert neural processes. Advances in neural information processing systems, 35: 2022.
Wang, Q.. Federici, M., and Hoof, H. Bridge the infer- gaps of neural processes via expectation maximiza- tion. arXiv preprint arXiv:2501 033244, 2025.
Wu, J., Zhang, C., Li, 7., Fu, H., Peng, X., and Zhou,
T. dugmatting: coomposed-uncertainty ting. arXiv preprini arXiv:2306.01452 2023.
mat-
Xiang.J. J.. Qiu, P., and Yang, Y. Fussmet: Fusing two sources of uncertainty for semi-supervised medical image seg- mentation. In International Conference on Medical Im- age Computing and Computer-Assisted Intervention, pp. 481-491. Springer, 2022.
Xu, Z., Yuan, B., Zhao, S., Zhang, Q., and Gao, X. Hier- archical point- based active learning for semi-supervised point cloud semantic segmentation. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 18098-18108, 2023.
Yue, Y., Mahadevan, S., Schult, J., Engelmann, Leibe, B., Schindler, K., and Kontogianni, T. Agile3d: Attention guided interactive multi-object 3d segmentation. arXiv preprint arXiv:2306.0009 00977, 2023.
Zhang, P., Wu, T., Sun, Li, W., and Su, Z. Refin- ing segmentation on-the-fly: An interactive framework for point cloud semantic segmentation. arXiv preprint arXiv:2403.c 06401, 2024.
Zhi, S., Sucar, E.,, Mouton, A., Haughton, I., Laidlow, T., and Davison, J. J. ilabel: Revealing objects neural fields. IEEE Robotics and Automation Letters, 8(2):832- 839, 2022.
Zhou, M., Wang, H., Zhao, Q., Li, Y., Huang, Y., Meng, D., and Zheng. Interactive segmentation gaussion process classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
pp.
Zhou, Y., Gu. J.. Chiang. T. Y., Xiang, F., and Su, H. Point- sam: Promptable 3d segmentation model for point clouds. arXiv preprint arXiv:2406.17744 2024.
Zhuang, C., Li, S., and Ding, H. Instance segmentation based 6d pose estimation industrial objects using point clouds for robotic sin-picking. Robotics and Computer- Integrated Manufacturing, 82:102441, 2023.

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
A. Derivation of ELBO
The proposed NPISEG3D formulated as:
p(Yr\Xr, Dc) / II P m=U
X X 2., z,)p(z"' [z. dzz plzz| (X)) dz,,
M
(12)
where p(zz X and p(zz" Zg. X denote the prior distributions of scene-specifer latent variable and each object- latent variables, respectively. Considering that X contains both click features and click label information, we substitute
De with X on the right-hand side of the above equation and throughout the following derivation. Then, the evidence lower bound is derived as follows:
log pl Y (c)
-log /
M
{
II
p P(Y XT, X X Ze)po(z" X²) dz ppo po(zz, (Xc) dz.
log
/
M
p P XT, X x , [Z,)
Po Zo. X
ddl
2,% 2s. X
dz
po(zs (x)g0(Zz Xx) dzg qo gz XT)
q6l (z
m=0
M
2 Eg (2, (xr)
log
X(Y X ,2,
;2,)
Pol 20
2s. X
Mah 2
2s. X
DKL. (expl//aee/XX///lo/ (c))
4a
(2.70
X
m=0
M
(x. (XT)
Eg (zm 12 X:r
log Y XT,
X x ,,, 22) DKL (2002 12.
m=0
-DKL (2kl/q0(22) (2. Xx) (loo (2 [x,]))
where (z(z, and (z." [z. X are variation posteriors of their corresponding latent variables, and é are parameters of inference modules for 24 and 2."h This ELBO effectively decomposes the log-likelihood into three terms: (1) reconstruction term, which maximizes the expected likeliioldo of the predictions Y under the latent variables; (2) KL divergence term for each object-specific latent variable z.", which aligns the approximate posterior with the prior; and (3) KL divergence term for the scene-specifee latent variable 2,, ensuring that the global context is appropriately captured. This formulation explicitly models uncertainty at both the scene and object levels, enabling the framework to effectively handle complex 3D segmentation tasks with both global and localized user interactions.
B. Additional Results.
B.1. Part Segmentation Results.
Methods
loU@51 t loU@10
loU@15
NoC@80 | NoC@85
NoC@90 |
InterObject3D
61.9
70.2
73.2
14.5
16.7
18.2
InterObject3D++
60.0
70.8
72.6
13.9
16.4
17.8
AGILE3D
59.6
70.1
74.4
14.5
16.7
18.0
NPISeg (Ours)
63.2
73.2
77.0
13.0
15.2
17.3
Table 8. Quantitative results part-wise segmentation using the PartNet dataset. The results are reported for multi-part segmentation tasks, where models are trained on ScanNet40. best-performing results are highlighted in bold. Our model consistently outperforms others in part segmentation accuracy.
To further evaluate the effectiveness of our method in part-wise segmentation, we conduct additional experiments PartNet under the multi-part interactive segmentation setting, where multiple parts of an object must be segmented simultancously.
For evaluation, we select six the most common objects with multiple parts, namely chairs, beds, scissors, sofas, tables, and lamps. ensure diversity the evaluation set, we randomly sample ten point clouds for each object category. This
12

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
setup allows assess the generalization ability of our model across different object geometries and part compositions. Since such selection involves randomness, we plan to release the object ids for fair comparison with method.
Table B.1 presents the quantitative results on the PartNet dataset for multi segmentation tasks. Our proposed NPISeg3D consistently outperforms existing methods, achieving the highest IoU across all evaluation thresholds (IoU@5, IoU@ 10, and @ 15), demonstrating its superior segmentation accuracy. Notably, NPISeg3D requires fewer user clicks to reach the desired IoU targets, as evidenced by the lowest @ 80, NoC@85, and NoC@90 values. These results indicate that our hierarchical probabilistic modeling effectively captures fine- gee-grained part-level details while maintaining global scene context, addressing the limitations of prior approaches like AGILE3D, which struggled with part-wise segmentation. The improved performance highlights the robustmess of NPISeg3D in handling complex multi-part segmentation scenarios with minimal user interaction. The qualitative results in Figure 6 further demonstrate effectiveness of our framework.
B.2. Benefits of Incorporating Previous Mask.
Methods
Datasets
loU@5 f
IoU@ 10
IoU@15
NoC@80
NoC@85 +
NoC 90
w/o Prev. Mask
89.0
90.5
91.0
2.9
4.6
7.8
S3DIS
w/ Prev. Mask
89.0
90.9
91.5
2.8
4.4
7.8
w/o Prev, Mask
43.5
47.8
52.3
16.6
17.3
17.8
KITTI360
w/ Prev. Mask
44.0
48.5
52.9
16.4
17.0
17.6
Table Renefit of incorporating previous mask into our framework. Results are reported for multi-object segmentation tasks, with models trained on the ScanNer40 dataset. Better results are highlighted in Bold.
In approach, leverage the previous segmentation mask to enhance the interactive segmentation process by incorpo- rating historical predictions into the current interaction. Specifically, the previous mask is first converted into one-hot representation, capturing class-wise information at each point. This one-hot encoding is then processed through learnable mask encoder, implemented as a four-layer Multi-Layer Perceptron (MLP), which outputs refined representation of size Nx15. The encoded mask features are subsequently concatenated with the point-wise features, providing enriched contextual information that facilitates more accurate predictions in subsequent interactions.
The results in Table 9 validate the effectiveness of incorporating the previous mask in both the S3DIS and KITTI360 datasets. The inclusion of previous mask information leads to consistently improved IoU scores across different click counts, demonstrating the model's ability to leverage past segmentation knowledge for better refinement. Moreover, the number of clicks required to reach target IoU thresholds (NoC metrics) is significantly reduced, highlighting the efficiency gains achieved through this strategy. These results affirm that utilizing prior segmentation masks effectively guides the model towards more precise and efficient interactive segmentation.
B.3. Ablation on Click Simulation Strategy during Training
Methods
Datasets
IoU@5
IoU@ 10
IoU@151 f
NoC@80 J
NoC@85 +
NoC@90 +
Iterative
89.3
91.1
91.7
2.9
4.5
7.8
S3DIS
RITM
89.0
90.9
91.5
2.8
4.4
7.8
Iterative
44.5
46.7
50.5
17.1
17.6
18.3
KITTI360
RITM
44.0
48.5
52.9
16.4
17.0
17.6
Table 10. Comparison of our model with different click simulation strategies during training. Results are reported for multi-object segmentation tasks, with models trained on the ScanNet40 dataset. Better results are highlighted in Bold.
AGILE3D (Yue et al., 2023) demonstrated the effectiveness of iterative training for interactive multi-object segmentation. However, iterative training incurs high computational costs, making it less practical for large-scale applications. In contrast, we adopt the more efficient RITM sampling strategy (Sofiiuk et al., 2022), which balances training efficiency and segmentation performance. See more analysis in Sec. D.2.
Table 10 compares our method using Iterative and RITM click simulation strategies during training on the S3DIS and KITTI360 datasets. For the S3DIS dataset, the Iterative method achieves slightly higher IoU scores across all click counts,
13

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
with loU@5 reaching 89.3 compared to RITM's 89.0. However, RITM exhibits a lower NoC@80 (2.8 vs. 2.9), suggesting that it requires fewer clicks achieve the same segmentation quality the 80% loU threshold. Both methods perform similarly in NoC@85 and NoC@90.
On the KITTI360 dataset, which poses greater challenges due to its outdoor LiDAR environment, the RITM strategy significantly outperforms the Iterative method in loU @ 10 and loU @ 15, with improvements of 1.5 and 2.4 points, respectively. Additionally, RITM achieves better NoC performance, indicating its superior efficiency in complex scenarios.
Overall, while the Iterative method slightly excels in high-quality indoor segmentation (S3DIS), RITM demonstrates better generalization challenging outdoor environments (KITTI360) by achieving higher IoU with fewer user interactions. Considering the balance between computation efficiency and performance, we adopt RITM as the click sampling strategy during our model training.
B.4. Computation Analysis
Model
Params/MB
FLOPs/G
Train Speed/days
Inference Speed/ms
Inter3D
37.86
0.47
80
Inter3D++
37.86
0.47
80
AGILE3D
39.30
4.73
6.4
60
NDISeg3D
40.00
4.94
2.7
65
Table 11. Computational analysis of NPISeg and state-of-the-art (SOTA) methods. The traing and inference speed are evaluated on a single NVIDIA A6000 GPU.
Table 11 presents comparative analysis of NPISeg3D and state-of-the-arr methods in terms model size, computational complexity, and efficiency. NPISeg3D has a slightly larger parameter size (40.00 MB) compared to AGILE3D (39.30 MB) and Inter3D variants (37.8) (37.86 MB), primarily due to the inclusion of the scene-level and object-level aggregators, as formulated in Eq. (6) and Eq. (7). Despite the increased FL.OPs (4.94G), NPISeg3I> achieves significantly faster training time of 2.7 days, outperforming AGILE3D's 6.4 days, attributed to the efficient random and iterative training strategy (see more details in Sec. D.2). At inference, NPISeg3D runs at 65 ms per scan scan-slightly slower than AGILE3D (60 ms) but substantially faster than Inter3D (80 ms). This demonstrates that NPISeg3D balances the added complexity probabilistic modeling with competitive runtime efficiency, making it well-suited for real-time applications.
Overall, these results confirm that NPISeg3D achieves favorable balance between computational efficiency and segmenta- tion performance, demonstrating its potential in practical applications.
B.5. Uncertainty Estimation
In our probabilistic segmentation framework, uncertainty estimation plays a crucial role in guiding user interactions and improving model reliability. To achieve this, we replace the mean operation used in segmentation logits computation (Eq. 11) with variance, leveraging multiple Monte Carlo samples to quantify the model's confidence. Specifically, our approach estimates uncertainty by measuring the variability across different latent samples. Since multiple click classifiers exist for each object, we further aggregate uncertainty by selecting the maximum value across all clicks, ensuring that the regions with the highest ambiguity are highlighted for further refinement.
Formally, the uncertainty map for object at each point is computed as:
No. cos (oos(XT, X Elcos XT, xm. )),
2
U
IIE 1E (EI...,.N N
(13)
N
1=1
where U denotes the uncertainty map for object m, No represents the number of click prototypes for object m, and B[]] denotes the expectation over Monte Carlo samples. The variance quantifies the spread of predictions across different latent samples, while the max operation identifies the highest uncertainty value among all click prototypes. The overall uncertainty map Ur is obtained by taking the maximum uncertainty across object classes, including the background, ensuring a
14

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
comprehensive representation of segmentation confidence across the entire scene. Formally, is computed
UT
max
T
(14)
This aggregated uncertainty map helps identify the most uncertain regions, guiding user interactions for more efficient refinement. This probabilistic formulation provides an interpretable measure of segmentation confidence, allowing the model identify regions requiring further corrective interactions. Such an approach enhances the interactive segmentation process by focusing user efforts on the most uncertain areas, thereby improving the overall segmentation quality in complex multi--bjeect 3D environments.
We provide visualization examples of uncertainty maps in Figure 7 and Figure 8.
Scene & GT
AGILE3D
NPISeg3D
S3DIS
3D
GT
1oU@6=69.7
10
IoU@66777.8
1oL@8=994.0
Replica
3D scene
GT
loU@44-78.4
1oU@6=81.0
loU@4-80.8
loU@6-84.9
KITT360
3D scene
GT
JoU@5-184 18.66
loU@ NOU@10-220 10.660066
IoU@5-422
IoU@ 10752.7
Figure Qualitative comparison between the state-of-the-art (SoTA) method AGILE3D and our proposed NPISeg3D on the interactive multi-object segmentation task. Newly added clicks are represented by dark-colored dots. Our NPISeg3D consistently achieves higher IoU scores with the same number of clicks, particularly on challenging outdoor LiDAR dataset KITTI-360.
Scene & GT
AGILE3D
NPISeg3D
S3DIS
3D scene
GT
loU@ 1-42.2
78..92
IoU@@3-86.9
loU@ 1-67.7
loUr22-80 60.6
10U@23-896
PartNet KITT1360
1oU@1-17.4
1oU182-49.3
1oU@3-4144
loU@1-200.8
loU@2-77.6
loU
3D scene
GT
1oU39 1oU82-3388
1oU@3-43.2
loU@1-21.7
1oU@2-57
IoU123-66.2
Figure 6. Qualitative comparison between the state-of-the-art (SoTA) method AGILE3D and our proposed NPISeg3D on the interactive single-object segmentation task. Newly added clicks represented by dark-colored dots. Our NPISeg3D consistently achieves higher IoU scores with the same number of clicks. Notably, our NPISeg3D also achieve decent segmentation performance on part-v wise segmentation PartNet dataset.
15

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
B.6. Qualitative Comparison with SoTA method.
In Figure 5 and Figure 6, we provide additional comparisons between the state-of-the-art (SoTA) method AGILE3D and proposed NPISeg3D the multi-object and single-object segmentation tasks, respectively. Generally, NPISeg3D consistently achieves higher IoU scores than AGILE3D with the number of clicks across diverse datasets. The improvement is particularly pronounced on the challenging outdoor LiDAR dataset KITTI-360. For example, in Figure 5, our NPISeg3D achieves an IoU of 42.4 after clicks, significantly outperforming AGILE3D's 18.6. These results further demonstrate the strong few-shot generalization derived from the NP framework with hicrarchical latent variable modeling.
B.7. More Qualitative Results.
We present more qualitative results regarding segmentation performance and uncertainty estimation in Figure 7 and Figure 8. As shown in these figures, our method not only demonstrates strong few-shot generalization capability-ian ie.. generating accurate segmentation masks with minimal user input-buu but also provides reliable and meaningful uncertainty estimation. Notably, high uncertainty concentrated around erroneous regions and object boundaries. This insight suggests that uncertainty maps could be leveraged guide the selection of click candidates for subsequent interaction rounds. We leave this exploration for future work.
C. User Study.
Toaee Joweeeicoo case 23 www.draaaeee
K
Backgound
Number
Figure User interface for interactive segmentation with integrated uncertainty estimation as guidance. the main window, provide input clicks, and the model generates a segmentation prediction. The upper-right window displays uncertainty estimation for the current segmentation, guiding users on where to focus in next interaction round.
To move beyond simulated user clicks and evaluate performance with real human interactions, we conduct user studies.
User Interface. To this end, we extend the user-friendly interface from (Yue et al.. 2023) incorporate uncertainty estimation. As shown in Figure the enhanced interface consists of two windows: a main window where users provide clicks and receive corresponding segmentation predictions, and sub-window in the upper-right corner that visualizes uncertainty estimation for the current segmentation, guiding users on where to focus in the next interaction round. The software is cross-platfor and browser-compatible supporting both interactive single- and multi-object segmentation. To enhance user interaction, various keyboard shortcuts are designed for efficiency. For example, Ctrl + Click designates a background region, while Number Click specifies an object. We appreciate the open-source tool provided by (Yue et al., 2023) and will release our code, along with the improved annotation tool, to facilitate future research.
16

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Ground Truth
loU@@ 1-62.9
@-
1oU05-8006
Uncertainty@11
Uncertainty@ @
Uncertainty@05
Ground Truth
1-
3D Sean
Uncertainty@11 1
Uncertainty@33
Uncertainty@5
Ground Truth
1oU51-558.
IoU@3-71.6
IoU@5-873
Scan
Uncertainty@@
Uncertainty@03
Uncertainty@5
Figure 7. Qualitative Results on Interactive Single-Object Segmentation. the multi-object segmentation task, IoU@ k denotes the average Intersection over Union (loU) after clicks per object. For clarity, only the first click per object is visualized, with subsequent clicks omitted. The uncertainty map highlights regions of model uncertainty, where brighter areas correspond to higher uncertainty. Notably, high uncertainty is predominantly localized to object boundaries and regions distant from user provided clicks.
17

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Ground Truth
1=78.2
loU@3=86.8
loU@5=90.1
3D Sean
Uncertainty@ @1
Uncertainty@3
Uncertainty@5 @
Ground Truth
loU@1=61.4
loU@3-71.6
IoU@5-75.2
3D Scan
Uncertainty@)
Uncertainty #3
Uncertainty 25
Ground Truth
loU@1=33.4
IoU@3=72.6
IoU@5=77.1
3D Scan
Uncertainty @1
Uncertainty@:
Uncertainty@5
Figure 8. More qualitative results on the interactive single-object segmentation task. multi-object segmentation task, loU@t denotes the average IoU after clicks per object. We show the first click per object, and omit following clicks for better visualization. In the uncertainty map, brighter regions indicate higher uncertainty. Our method provides reliable uncertainty estimation, with high uncertainty primarily focused object edges and far from clicks.
18

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Design. We conducted user study involving 10 participants with no prior experience with point cloud annotation. Firstly, each participant was assigned the task to annotate 4 scenes, where two of the scenes from the S3DIS dataset and the other two from the KITTI-360 dataset. Each of these scenes consists of 4-7 objects, which cover various categories, including chairs, tables, cars, etc. Participants were allowed to refine the segmentation results step by step based on their own preferences, rather than being constrained to selecting the regions with the highest error as the next click candidate, as done in simulation. Secondly, during annotation, each participant continued annotating until they were satisfied with the results. Throughout the process, we recorded the average intersection-vve-uniiis (loU) score and the number of clicks. Finally, we computed the average of these metries across all participants and present the results in Table 7.
D. Experimental Setup Details.
D.1. Dataset Details.
We adopt the experimental setting from prior work (Yue et al., 2023) and train our model on the ScanNetV2- Train dataset (Dai et al., 2017), which contains 1,200 indoor scenes. Evaluation is conducted across four diverse datasets, covering both indoor and outdoor environments: ScanNetV2-Val (Dai et al., 2017), S3DIS (Armeni et al., 2016), Replica (Straub et al., 2019), and KITTI-360 (Liao et al., 2022). Additionally, we further evaluate the part-level segmentation capability of our NPISeg3D and existing models on the PartNet dataset (Mo et al., 2019).
ScanNetV2 (Dai al., 2017) consists of richly annotated indoor scenes, serving the primary dataset for training and validation. S3DIS (Armeni et al., 2016) features 272 indoor scenes across six large areas, presenting significant domain gap from Ve.. Replica (Straub et al., 2019) is a photorealistic indoor dataset comprising high-quality 3D reconstructions of real-world environments, offering diverse scene layouts and materials for testing model generalization. KITTI-360 (Liao et al., 2022), an outdoor LiDAR point cloud dataset, is designed for 3D perception tasks in autonomous driving and provides large-scale outdoor scenes.
PartNet (Mo et al., 2019) focuses on part-level segmentation, offering hierarchical annotations of 3D objects. This dataset challenges the model's 's ability to perform fine- fine-grained segmentation by decomposing objects into semantic subcomponents. These datasets collectively provide diverse and comprehensive benchmark to evaluate the robustness and generalizability of our model across varying domains and tasks.
D.2. Implementation Details.
Detailed Model Setting. In NPISeg3D, the point encoder consists of a backhone network and an attention network to effectively process 3D point cloud data. Specifically, we adopt the Minkowski Res 16UNee33 (Choy et al., 2019) backbone, following prior works (Kontogianni et al., 2023; Yue et al., 2023; Schult et al., 2023). The 3D scene is first quantized into N' spare voxels with fixed resolution of 5cm, ensuring efficient and consistent representation, as done in previous studies (Kontogianni et al., 2023; Yue et al., 2023; Schult et al., 2023). The backbone processes the sparse voxelized input and outputs feature map of size N' x 96, which is further projected to 128 channels by 1 1 convolution. Following (Yue et al., 2023), we employ attention network composed of multiple layers that facilitate interaction between click and scene features. These layers include fusion of user-provided inputs with the global scene context to enhance segmentation performance.
scene-to-click, and click-to-click attention mechanisms, enabling effective
Model Training. We train NPISeg3D end-to-end for 600 epochs using the Adam optimizer with an initial learning rate of 0.0005. The learning rate reduced by a factor of 0.1 after 500 epochs to facilitate convergence. Training is performed on a single Tesla A6000 GPU with a batch size of 5. The KL loss coefficient Again Eq. (10) is set to 0.005. For the segmentation loss Laeg, we use combination of cross-entropy loss and dice loss, with coefficients of 1 and 2, respectively.
Click Simulation Strategy for Training, In interactive 3D segmentation, click simulation during training plays crucial role in aligning the model's behavior with real-world user interactions. The multi- iterative training strategy proposed in (Yue et al., 2023) has been demonstrated to effectively improve segmentation performance by simulating user clicks iteratively based on the model's predictions from previous rounds. This iterative approach closely mimics the test-time click sampling strategy, enhancing the model's ' ability to refine segmentation with progressive user feedback.
However, despite its effectiveness, this iterative simulation strategy is computationally expensive and time-consuming (Sofi- iuk et al., 2022), as evidenced by the runtime analysis in Table 11. The high computational cost makes it impractical for large-scale training scenarios. In contrast, interactive 2D segmentation commonly employs more efficient strategy, known
19

---

Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Random and Iterative Training Mechanism (RITM) (Sofiiuk al., 2022), which strikes balance between training efficiency and performance. RITM initializes the training process by randomly sampling initial set of clicks, followed by limited number of iteratively sampled clicks refine segmentation predictions.
Inspired by RITM, we extend its application to the interactive 3D segmentation task, including the multi-object segmentation setting. Specifically, our approach combines random and iterative sampling strategies to optimize both efficiency and accuracy. First, an initial set user clicks is randomly sampled, providing diverse training samples across different objects. Then, a variable number of iterative clicks, ranging from to Neter are added progressively based on the model's prediction errors, ensuring better alignment with real-world interaction scenarios.
Click Simulation Strategy for Inference. Following prior works (Kontogianni et al., 2023; Yue et al., 2023; Zhang et al., 2024), we adopt standardized automated click simulation strategy to ensure reproducible evaluation. Specifically, the first click is placed at the center of the object within the ground truth mask. Subsequent clicks are iteratively placed at the centroid the largest misclassified region by comparing the predicted segmentation with the ground truth mask. This process continues until the segmentation reaches the maximum click limit reached, i.e., totally 20 clicks.
20


Reviews:
[{"review_id": "TIgW5RPCPT", "reviewer": "Reviewer_F92a", "summary": "This paper proposes a method using neural processes for 3D interactive segmentation which in addition to segmentations, also enables uncertainty estimations. The proposed method uses a hierarchical latent structure to capture both local and global concepts and a probabilistic prototype modulator which allows for the model to have better object-aware context for its predictions. The paper validates its claims by comparing its method to other interactive 3D segmentation approaches showing improved performance for both single and multi-object segmentation and demonstrating improved generalization abilities over existing methods. Additionally, thorough ablations validate the importance of key method components (such as the hierarchical latent variables and the uncertainty estimation).\n\n## Update after rebuttal\nAfter reading the rebuttal, my concerns regarding further qualitative comparisons were addressed as more examples are shown in the supplemental material. However, I still think the paper would benefit form showing results on more scenes and more diverse scenes which could perhaps be added to the supplemental material. My other concern regarding generalization was addressed by the fine-tuning metrics presented in the rebuttal. While these results are convincing, it would help to include more detail on how much compute this fine-tuning requires. Additionally, it is not clear how feasible fine-tuning will be in real would applications. Having read the rebuttal, I am maintaining my score of weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: Include discussion of some interactive 3D segmentation methods that translate SAM features into 3D such as SAM3D and SA3D.***\n\nThank you for suggesting these relevant interactive 3D segmentation works, SAM3D and SA3D, both of which explore translating 2D SAM features into 3D. Specifically, SAM3D leverages the pretrained SAM to automatically generate 2D masks in RGB images, and then maps these masks into 3D point cloud using pixel-wise depth from RGB-D images—without requiring additional training or fine-tuning. Similarly, SA3D utilizes radiance fields as an off-the-shelf prior to bridge multi-view 2D images and 3D space. It first generates a 2D mask using SAM on a single view, and then performs mask inverse rendering and cross-view self-prompting to iteratively refine the 3D mask of the target object across multiple views.\n\nIn contrast, our approach directly tackles native 3D interactive segmentation by taking 3D interactive clicks and point cloud inputs without relying on 2D projections. We view SAM3D, SA3D, and our method as complementary contributions that collectively advance 3D interactive segmentation. \n\n***We will incorporate a discussion of these works in the final version*** to provide a more comprehensive perspective on related approaches.\n\n***Q2: Supplementary materials issues.***\n\nWe kindly clarify that the ***Appendix (supplementary materials) was included in the submission (pp. 12–20)***. In Appendix, we provided the ELBO derivation (Sec. A), additional quantitative and qualitative results (Sec. B) to show the effectiveness of our method across different settings, user study details (Sec. C) to demonstrate the usability in practical scenarios, and implementation details (Sec. D) such as model structure and click simulation strategy.\n\n***Q3: More qualitative results on diverse inputs.***\n\nThanks for this valuable comment. ***In Appendix (pp. 12–20), we provided extensive qualitative comparisons in Figures 4–7, showcasing diverse input scans and click prompts.*** Specifically, Figure 4 illustrates segmentation masks generated with varying numbers of clicks in the single-object setting, while Figure 5 presents results for multi-object segmentation. Figures 6 and 7 further display segmentation masks and uncertainty maps across different click counts for single- and multi-object cases. These examples cover challenging scenarios involving varying object complexities and sparse user inputs, highlighting the robustness and generalization capability of our method.\n\n***Q4: Performance falls short on challenging out-of-domain datasets like KITTI-360.***\n\nThank you for your feedback. Similar to prior methods such as AGILE3D [1] and Inter3D [2], our model is trained exclusively on ScanNet and evaluated on out-of-domain datasets like KITTI-360 to evaluate generalization. The substantial domain gap between ScanNet (indoor RGB-D scenes) and KITTI-360 (outdoor LiDAR scenes) inevitably leads to suboptimal performance on KITTI-360. As a result, even SOTA method like AGILE3D achieves only 44.4% mIoU after 5 clicks under the single-object setting.\n\nNevertheless, our method demonstrates significant improvements over previous SOTA approaches. Specifically, our NPISeg3D improves mIoU by 10.9% and 9.8% over the previous SoTA AGILE3D on KITTI-360 under the single-object and multi-object settings, respectively, highlighting its effectiveness even under large distribution shifts.\n\n|               | **mIoU@5** | **mIoU@10** | **mIoU@15** | **NoC@80** | **NoC@85** | **NoC@90** |\n|---------------|------------|-------------|-------------|------------|------------|------------|\n| **w/o fine-tuning** | 44.0       | 48.5        | 52.9        | 16.4       | 17.0       | 17.6       |\n| **w/ fine-tuning**  | 79.2       | 82.8        | 85.4        | 8.5        | 10.8       | 12.3       |\n\nTo enable effective deployment in downstream tasks, one feasible solution is to consider domain adaption techniques, e.g, domain-specific fine-tuning. As shown in the table above, our model—when fine-tuned on KITTI-360—achieves significantly better performance across all metrics, approaching levels that are practical for downstream tasks. For example, mIoU@5 improves from 44.0% to 79.2%, and the number of clicks (NoC@90) required to reach 90% IoU decreases from 17.6 to 12.3. These results demonstrate the strong adaptability of our method to specific domains when fine-tuning data is available.", "claims_and_evidence": "The claims made are supported by sufficient evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed make sense for the given task. The authors evaluate their method on multiple segmentation datasets for both single and multiple objects as well as provide results as compared to user generated segmentations.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design appears sound.", "relation_to_broader_scientific_literature": "This paper approaches the task of interactive 3D segmentation using NPs and predicts the segmentations in a probabilistic manner enabling uncertainty predictions which are not supported by existing interactive 3D segmentation methods such as InterObject3D.", "essential_references_not_discussed": "While not strictly necessary the paper might benefit from discussion of some interactive 3D segmentation methods that translate SAM features into 3D such as SAM3D[2] and SA3D[3].\n\nReferences:\n[1] Kirillov, Alexander, et al. \"Segment anything.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n[2] Yang, Yunhan, et al. \"Sam3d: Segment anything in 3d scenes.\" arXiv preprint arXiv:2306.03908 (2023).\n[3] Cen, Jiazhong, et al. \"Segment anything in 3d with nerfs.\" Advances in Neural Information Processing Systems 36 (2023): 25971-25990.", "other_strengths_and_weaknesses": "Strengths:\n- Improved performance over existing methods for interactive 3D segmentation\n- Enables uncertainty predictions by using NP framework, a capability that is not supported by existing methods for this task\n\nWeaknesses:\n- Not enough qualitative comparisons, the paper would benefit from showing more qualitative results on diverse inputs. Perhaps this could be included in supplementary material if there is not enough space in the main paper.\n- As discussed in the paper, while the generalization shown is an improvement over existing methods, it is not necessarily successful enough on challenging datasets such as KITTI-360 to be useful for downstream tasks.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CTc17jAEPk", "reviewer": "Reviewer_TzH1", "summary": "This paper presents NPISeg3D, a novel probabilistic framework for interactive 3D segmentation based on neural processes (NPs), which addresses the key challenges of generalizing from sparse user clicks and quantifying predictive uncertainty. The framework introduces a hierarchical latent variable structure and a probabilistic prototype modulator to enhance few-shot generalization and provide reliable uncertainty estimation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: Computational efficiency and Reliability of uncertainty estimation.***\n\nThank you for your valuable comments. Below, we address each aspect in turn.\n\n***Computational efficiency.*** Our  NPISeg3D introduces negligible extra parameters introduced by our neural process module which enhances generalization and enables reliable uncertainty modeling. As shown in Table 11 of the Appendix (pp. 14), NPISeg3D has 40.0MB of parameters, compared to 39.3MB in the previous SoTA AGILE3D. This amounts to only a 1.8% increase, which is minimal.\n**Despite this slight overhead, NPISeg3D delivers significantly improved segmentation performance**. For instance, It surpasses AGILE3D by 11.3% mIoU with 5 clicks under the single-object setting on KITTI-360, demonstrating that **the modest computational cost is well justified by the substantial performance gains**.\n\n***Reliability of uncertainty estimation.*** To evaluate the reliability of our model’s uncertainty estimation, we provide extensive qualitative results in Figures 6 and 7 of the supplementary material. In particular, the first two rows of Figure 6 show that **after a single click, the uncertainty map effectively highlights erroneous regions and object boundaries—areas that are inherently ambiguous in interactive segmentation**. These results demonstrate that our method produces meaningful and reliable uncertainty estimates even with very few clicks. Moreover, as the number of clicks increases, we observe a clear reduction in uncertainty alongside improved segmentation quality, further validating the robustness and interpretability of the predicted uncertainty. \n\n***Q2: Extension to other 3D representations, such as 3DGS. And potential integration with open-vocabulary 3DGS semantic segmentation methods, such as GOI and Chatsplat.***\n\nWe appreciate the reviewer’s insightful comment. **Our current method is primary designed for point cloud inputs**, which provide a simple and efficient representation that generalizes well across diverse scenes without requiring scene-specific optimization. \n\nMeanwhile, we also agree that extending interaction to other 3D representations, such as 3D Gaussian Splatting (3DGS), is a highly promising direction. However, direct interactive segmentation in 3DGS is non-trivial: individual Gaussians lack explicit semantic meaning and are optimized per scene, making object-level interaction challenging. Recent works, such as Click-Gaussian [1], GaussianCut [2], and ISRF [3], have explored propagating 2D user interactions into 3D via dense multi-view supervision. These approaches typically rely on 2D-view-based interactions and project multi-view masks into 3D space.\n\nInspired by these methods, **our method could be extended to such settings by first generating 2D masks from user clicks using our probabilistic framework, i.e., hierarchical neural process structure, then lifting these masks into 3DGS space using known camera poses or depth maps**. This could enable selecting or reweighting Gaussians based on segmented regions, facilitating interactive segmentation without requiring per-Gaussian semantic labels or retraining.\n\nRegarding integration with open-vocabulary 3DGS segmentation, methods such as GOI and ChatSplat incorporate language-guided semantics into 3DGS. We believe combining interactive inputs (e.g., clicks or referring expressions) with open-vocabulary reasoning is a compelling future direction. For instance, **user interactions could guide the adaptation of semantic hyperplanes or modulate Gaussian importance weights during inference**.\n\n**Although 3DGS and other 3D representations are not the main focus of our current work, we will include the above discussions in the final version to provide a broader perspective.** We also find it an interesting direction to explore how our probabilistic interactive framework could be adapted to 3DGS pipelines to support language-driven 3D interaction in future work.\n\n[1] Choi, Seokhun, et al. \"Click-gaussian: Interactive segmentation to any 3d gaussians.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[2] Jain, Umangi, Ashkan Mirzaei, and Igor Gilitschenski. \"GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024. \n\n[3] Goel, Rahul, et al. \"Interactive segmentation of radiance fields.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The paper conducts experiments on multiple datasets, including ScanNetV2, S3DIS, Replica, and KITTI-360, covering both indoor and outdoor environments, ensuring a certain degree of representativeness.", "theoretical_claims": "Most of them are reasonable, and I haven't checked all the formulas.", "experimental_designs_or_analyses": "Comprehensive comparative experiments are carried out with existing methods such as InterObject3D, InterObject3D++, and AGILE3D, demonstrating the superiority of NPISeg3D in segmentation accuracy and user interaction efficiency.", "relation_to_broader_scientific_literature": "The 3D interactive segmentation studied in this work can, to some extent, be used for constructing three-dimensional datasets in the medical field.", "essential_references_not_discussed": "Most of the relevant literature has already been discussed.", "other_strengths_and_weaknesses": "The paper is well-written and relative easy to follow.\n\nNPISeg3D has a slightly larger parameter size than AGILE3D and Inter3D variants, which may restrict its application in scenarios with limited computing resources. While NPISeg3D provides uncertainty estimates, their reliability may need further verification and enhancement, especially with very few clicks.", "other_comments_or_suggestions": "The current model takes point clouds as input. I hope users can discuss the applications of other 3D representations in the article, such as 3D Gaussian Splatting (3DGS). Specifically, whether it is possible to achieve interactive 3DGS segmentation and the potential integration with open-vocabulary 3DGS semantic segmentation methods, such as  GOI [1] and Chatsplat [2].\n\n\n[1] Goi: Find 3d gaussians of interest with an optimizable open-vocabulary semantic-space hyperplane\n[2] ChatSplat: 3D Conversational Gaussian Splatting", "questions_for_authors": "Reference to “Other Comments Or Suggestions”", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No ethical concerns"}}, {"review_id": "0Vk0rcRUrg", "reviewer": "Reviewer_E5kU", "summary": "Main Contributions & Findings\nThe paper introduces NPISeg3D, a novel probabilistic framework for interactive 3D segmentation, leveraging Hierarchical Neural Processes (NPs) to tackle two key challenges:\n1. Few-shot generalization – enabling accurate segmentation from sparse user clicks.\n2. Uncertainty estimation – providing reliable confidence measures to guide user interactions.\nKey findings include:\n* NPISeg3D achieves superior segmentation performance with fewer user clicks compared to state-of-the-art (SoTA) baselines.\n* It improves generalization in both in-domain and out-of-domain settings.\n* The probabilistic framework enables explicit uncertainty quantification, enhancing interpretability.\nMain Algorithmic/Conceptual Ideas\n1. Hierarchical Neural Processes (NPs):\n    * Introduces a scene-specific latent variable (for capturing global scene context) and object-specific latent variables (for modeling fine-grained object characteristics).\n    * These hierarchical latent variables improve generalization and facilitate probabilistic modeling of segmentation tasks.\n2. Probabilistic Prototype Modulator:\n    * Dynamically adjusts click-based segmentation prototypes using learned object-specific latent variables.\n    * Improves adaptability to user interactions and enhances uncertainty estimation.\n3. Probabilistic Formulation:\n    * The model treats user clicks as context data and remaining 3D points as target data in a probabilistic setting.\n    * Uses variational inference to estimate segmentation probabilities and model uncertainties.\n4. Efficient Training & Inference:\n    * The model employs variational inference with an evidence lower bound (ELBO) to optimize segmentation accuracy while maintaining robust uncertainty estimates.\n    * At inference, segmentation masks are generated using a Monte Carlo-based probabilistic approach.\nMain Results\n* Quantitative Performance:\n    * NPISeg3D outperforms AGILE3D, InterObject3D, and other SoTAs on multiple 3D segmentation benchmarks, including ScanNet, S3DIS, Replica, and KITTI-360.\n    * In out-of-domain settings, NPISeg3D consistently achieves higher IoU (Intersection over Union) with fewer user interactions.\n    * Reduces the number of clicks needed to achieve high accuracy (e.g., reducing NoC@80 on KITTI-360 from 17.4 (AGILE3D) to 16.4).\n* Qualitative & User Study Findings:\n    * NPISeg3D provides more precise segmentation masks with fewer clicks.\n    * Generates uncertainty maps that highlight unreliable regions, guiding further user interaction.\n    * Real-user experiments confirm that the model effectively improves annotation efficiency and accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: Limitations in Out-of-Domain Generalization.  (1) Out-of-domain performance falls short in-domain. (2) The paper does not explore domain adaptation techniques.***\n\nThank you for your insightful comments. Similar to previous methods like AGILE3D, our model is trained solely on ScanNet and evaluated on out-of-domain datasets such as KITTI-360 to assess its generalization capability. **Due to the significant domain gap between ScanNet (indoor RGB-D scenes) and KITTI-360 (outdoor LiDAR scenes), segmentation on KITTI-360 remains inherently challenging.** As a result, even SOTA method like AGILE3D achieves only 44.4% mIoU after 5 clicks under the single-object setting.\n\nNevertheless, NPISeg3D achieves substantial improvements over previous state-of-the-art (SoTA) methods on KITTI-360. Specifically, it improves mIoU by 10.9% and 9.8% over AGILE3D under the single-object and multi-object settings, respectively, demonstrating the robustness of our approach even under large distribution shifts. \n\nTo further enhance out-of-domain performance, one feasible solution is to consider domain adaption techniques, e.g, domain-specific fine-tuning. As shown in the table below, **our model—when fine-tuned on KITTI-360—achieves significantly better performance across all metrics, approaching levels that are practical for downstream tasks.** For example, mIoU@5 improves from 44.0% to 79.2%, and the number of clicks (NoC@90) required to reach 90% IoU decreases from 17.6 to 12.3. These results demonstrate the strong adaptability of our method to specific domains when fine-tuning data is available.\n\n|               | **mIoU@5** | **mIoU@10** | **mIoU@15** | **NoC@80** | **NoC@85** | **NoC@90** |\n|---------------|------------|-------------|-------------|------------|------------|------------|\n| **w/o fine-tuning** | 44.0       | 48.5        | 52.9        | 16.4       | 17.0       | 17.6       |\n| **w/ fine-tuning**  | 79.2       | 82.8        | 85.4        | 8.5        | 10.8       | 12.3       |\n\nWe agree that domain adaptation is a valuable and complementary direction for interactive 3D segmentation,  and  will consider more advanced domain adaptation techniques such as parameter-efficient fine-tuning and test-time adaption in future research.\n\n***Q2: Computational Complexity & Scalability.***\n\nAs shown in Table 11 of the Appendix, our method introduces only marginal computational overhead due to its probabilistic modeling. **The parameter size increases by just 1.8% (40.00MB vs. 39.30MB), and the FLOPs remain comparable (4.94G vs. 4.73G).** Although Monte Carlo sampling with 5 samples adds some overhead, both sampling and decoding are fully parallelized. As a result, our method achieves an inference speed of 65 ms per forward pass, closely matching AGILE3D’s 60 ms and supporting real-time interaction.\n\nMoreover, our iterative and random training strategy accelerates training by approximately 2.5× compared to AGILE3D, further highlighting the scalability of our approach for large-scale datasets.\n\n**Despite this slight overhead, our method delivers significant performance improvements**. For instance, on the high-resolution LiDAR dataset KITTI-360, it surpasses AGILE3D by 11.3% mIoU with just 5 clicks under the single-object setting, demonstrating that **the modest computational cost is well justified by the substantial performance gains.**\n\n***Q3: Clarity & Accessibility of Technical Concepts. (1) Rigorous math of Neural Processes may limit accessibility for non-experts. (2). The explanation of the hierarchical NP structure could be improved.***\n\nThank you for the helpful suggestion. We fully agree that it is important to make the mathematical formulation more intuitive and accessible, especially for readers who are less familiar with Neural Processes (NPs). **In the final version, we will revise the exposition to improve clarity and incorporate more intuitive explanations where appropriate.**\n\n***To better illustrate the hierarchical NP structure, we further provided a graphical model at the following anonymous link:*** https://github.com/anonusers2025/rebuttal_figure/blob/main/graphical_model.pdf. This figure offers a more visual and accessible explanation of how our model integrates scene-level and object-level latent variables for probabilistic interactive 3D segmentation. We will further improve and refine this part in the final version to enhance readability and understanding.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "The key contributions of NPISeg3D build upon and extend several areas of research in interactive 3D segmentation, probabilistic modeling, and neural processes. Below is a structured discussion of how its contributions relate to the broader scientific literature.\n\n1. Interactive 3D Segmentation\nRelated Work:\n* Interactive segmentation has been explored in both 2D and 3D domains, with works like InterObject3D (Kontogianni et al., 2023) and AGILE3D (Yue et al., 2023) focusing on multi-object segmentation in point clouds.\n* CRSNet (Sun et al., 2023) used click-based simulation to refine segmentation masks iteratively, but lacked probabilistic uncertainty modeling.\n* SemanticPaint (Valentin et al., 2015) introduced interactive 3D labeling using real-time feedback but relied on handcrafted features rather than learning-based models.\nNPISeg3D's Novelty:\n* Unlike prior deterministic models, NPISeg3D is the first to introduce a probabilistic framework into interactive 3D segmentation.\n* It improves few-shot generalization, reducing the number of user clicks required for accurate segmentation.\n* Uncertainty estimation is incorporated, which was missing in prior interactive segmentation methods.\n\n\n2. Few-shot Learning & Neural Processes\nRelated Work:\n* Neural Processes (NPs) (Garnelo et al., 2018) introduced a probabilistic approach to function approximation, learning to model distributions over functions with minimal supervision.\n* Conditional Neural Processes (CNPs) (Garnelo et al., 2018) extended NPs by conditioning outputs on observed context points.\n* Attentive Neural Processes (ANPs) (Kim et al., 2019) improved information aggregation by integrating attention mechanisms.\n* NP-based approaches have been used in continual learning (Jha et al., 2024) and semi-supervised learning (Wang et al., 2023) but had not yet been applied to interactive segmentation.\nNPISeg3D's Novelty:\n* It formulates interactive segmentation as a probabilistic function approximation problem, leveraging hierarchical neural processes to improve generalization from limited user inputs.\n* The hierarchical latent variable structure (scene-level and object-level latent variables) extends standard NPs, making them more effective for structured segmentation tasks.\n* It introduces a probabilistic prototype modulator, which enhances adaptability to new objects in few-shot scenarios.\n\n3. Uncertainty Estimation in Segmentation\nRelated Work:\n* Uncertainty estimation has been widely studied in Bayesian Deep Learning (Gal & Ghahramani, 2016) and Monte Carlo Dropout (MC Dropout) (Xiang et al., 2022).\n* Deep Gaussian Processes (DGPs) (Jakkala, 2021) modeled uncertainty in deep learning by capturing distributions over functions.\n* Uncertainty-aware methods have been applied in medical imaging (Rakic et al., 2024) and autonomous driving (Michelmore et al., 2020), where error quantification is crucial.\n* Existing segmentation models such as AGILE3D (Yue et al., 2023) and InterPCSeg (Zhang et al., 2024) neglected uncertainty estimation.\nNPISeg3D's Novelty:\n* It directly incorporates predictive uncertainty into the segmentation pipeline, allowing users to identify unreliable regions.\n* Unlike MC Dropout, which samples from model weights, NPISeg3D models structured uncertainty via latent space sampling.\n* It outperforms MC Dropout-based approaches, as shown in ablation studies.\n\n\n4. Multi-object and Multi-modal Segmentation\nRelated Work:\n* Multi-object segmentation approaches like AGILE3D (Yue et al., 2023) used attention-based mechanisms for segmenting multiple objects.\n* OpenMask3D (Takmaz et al., 2023) introduced open-vocabulary segmentation in 3D, but lacked interactivity.\n* PointSAM (Zhou et al., 2024) proposed prompt-based 3D segmentation, leveraging large vision models, though it does not incorporate user feedback.\nNPISeg3D's Novelty:\n* Unlike AGILE3D, which relies on deterministic attention mechanisms, NPISeg3D introduces hierarchical latent variables to model inter-object relationships probabilistically.\n* Unlike OpenMask3D, NPISeg3D is not restricted to pre-defined object categories and instead adapts dynamically to user inputs.\n\n5. Interactive Machine Learning & Human-in-the-Loop AI\nRelated Work:\n* Human-in-the-loop (HITL) learning has been used in areas like active learning (Xu et al., 2023) and iterative annotation frameworks (Sofiiuk et al., 2022).\n* SemanticPaint (Valentin et al., 2015) allowed users to refine 3D segmentations iteratively.\n* Interactive semi-supervised segmentation (Wang et al., 2022) explored how user inputs could improve segmentation accuracy.\nNPISeg3D's Novelty:\n* It integrates HITL learning with probabilistic uncertainty modeling, enabling more efficient human-in-the-loop correction.\n* Unlike existing interactive segmentation frameworks, it prioritizes regions with high uncertainty, guiding user inputs more effectively.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n1. Originality & Novelty\n    * The paper presents the first probabilistic framework for interactive 3D segmentation, which is a significant departure from deterministic methods like AGILE3D and InterObject3D.\n    * The hierarchical neural process (NP) formulation is an innovative adaptation of NPs to segmentation tasks, particularly in an interactive, few-shot setting.\n    * The probabilistic prototype modulator is a novel mechanism that dynamically refines click prototypes, improving both segmentation performance and uncertainty estimation.\n2. Significance & Impact\n    * The proposed method has strong potential applications in real-world domains such as autonomous driving, robotic perception, and medical imaging, where segmentation reliability is critical.\n    * The integration of uncertainty quantification in an interactive framework could fundamentally change user interaction strategies in 3D annotation, making segmentation more adaptive and efficient.\n    * Demonstrates strong few-shot generalization capabilities, making it highly applicable in low-data scenarios, a common challenge in real-world applications.\n3. Empirical Rigor & Comprehensive Evaluation\n    * The paper evaluates NPISeg3D across four benchmark datasets (ScanNet, S3DIS, Replica, KITTI-360), including both in-domain and out-of-domain settings, demonstrating its robustness.\n    * The quantitative comparisons against strong baselines (InterObject3D, AGILE3D) show consistent superiority in terms of IoU and click efficiency (NoC@80, NoC@85, NoC@90).\n    * The user study strengthens the claim that NPISeg3D enhances annotation efficiency and provides practical benefits in real-world interactive segmentation tasks.\n4. Clarity & Reproducibility\n    * The paper is generally well-written with clear explanations of the methodology, probabilistic formulation, and hierarchical latent variable design.\n    * The detailed ablation studies (evaluating latent variables, modulation strategies, uncertainty modeling, etc.) provide insights into why each component is effective.\n    * Mathematical formulations are rigorous and systematically derived, making the theoretical contributions accessible to the reader.\n\nWeaknesses\n1. Limitations in Out-of-Domain Generalization\n    * Despite its strong performance, NPISeg3D still lags behind in-domain performance when applied to out-of-domain datasets, such as KITTI-360.\n    * The paper does not explore domain adaptation techniques, which could further improve generalization to unseen datasets.\n2. Computational Complexity & Scalability\n    * The probabilistic inference framework introduces additional computational overhead compared to deterministic models like AGILE3D.\n    * The need for Monte Carlo sampling for latent variable inference could slow down real-time interaction, particularly in large-scale datasets.\n    * The scalability of NPISeg3D to very large point clouds (e.g., in high-resolution LiDAR-based perception) remains unclear.\n3. Clarity & Accessibility of Technical Concepts\n    * While mathematically rigorous, the paper assumes a high level of familiarity with Neural Processes (NPs), which may limit accessibility for non-experts.\n    * The explanation of the hierarchical NP structure could be improved with more intuitive visualizations or concrete examples.\n\nOverall, I believe this paper is highly valuable, making significant contributions to the field of interactive 3D segmentation through its novel probabilistic framework. The weaknesses I have mentioned are not fundamental flaws but rather aspects where further discussion could enhance the clarity, applicability, and impact of the work.", "other_comments_or_suggestions": "NO", "questions_for_authors": "Please see my detailed comments above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WyB5ye5nfL", "reviewer": "Reviewer_uwh8", "summary": "This paper addresses the problem of interactive 3D segmentation, where the model segments target objects based on positive and negative user clicks. This paper proposes a probabilistic framework built upon Neural Processes (NPs) to enhance model generalisation. Specifically, the model aggregates object embeddings into scene-level embeddings to capture global context, and subsequently updates object embeddings to represent object-specific characteristics. The method is evaluated on several benchmarks and demonstrates improved generalisation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: Architecture details for reproduction and code for clarification.***\n\nThanks for your valuable suggestions. In Appendix D.2 (pp. 19), we provided additional architectural details of our framework. Specifically, following AGILE3D, the point encoder in Figure 1 consists of a backbone—Minkowski Res16UNet34C—along with an attention module for click-scene interaction. The architectural components related to Neural Processes (NPs) are depicted in Figure 1 and detailed in Sec. 4.2–4.4, including details about scene-level and object-level aggregators (Sec.4.2), probabilistic prototype modulator (Sec.4.3), and training objectives (Sec.4.4).\n\nTo better illustrate the hierarchical NP structure, we further provided a graphical model at the following anonymous link: https://github.com/anonusers2025/rebuttal_figure/blob/main/graphical_model.pdf. This figure offers a more visual and accessible explanation of how our model integrates scene-level and object-level latent variables for probabilistic interactive 3D segmentation. \n\n***We will further improve  the description about architecture details in the final version*** to facilitate reproduction and understanding. ***We will also release our code*** to facilitate implementation and future research.\n\n***Q2: Discuss and analyze the reasons behind minor improvements on the ScanNet.***\n\nWe thank the reviewer for the comment. Our model is exclusively trained on ScanNet, and we observe relatively modest improvements over AGILE3D on this benchmark. This is expected for several reasons.\n \nFirst, all baseline models and ours are trained on ScanNet, and these models have already seen a large number of structurally similar indoor scenes during training, the segmentation task becomes relatively easy in this in-domain setting. As a result, strong baselines like AGILE3D already perform very well with only a few user clicks, e.g., reaching 82.3% IoU after 5 clicks, and the performance quickly saturates. Under such conditions, the benefits of our probabilistic design are less pronounced, and the potential for further improvement is naturally limited.\n\nSecond, our method is specifically designed to handle uncertainty and model generalization through a probabilistic framework with hierarchical latent variables. These advantages are less evident in in-domain settings such as ScanNet, where the model benefits from exposure to similar scenes during training. In contrast, they become significantly more valuable when applied to challenging, unseen, and out-of-domain scenarios. For example, on KITTI-360, which features unstructured outdoor LiDAR scans with large domain gaps, NPISeg3D achieves +10.9% and +9.8% mIoU improvement over AGILE3D under single-object and multi-object settings, respectively. Similar trends are observed on S3DIS and Replica.\n\nThese results highlight that ***while our method shows modest gains in in-domain settings like ScanNet, it delivers substantial benefits under more realistic, ambiguous, and out-of-domain conditions—where generalization and uncertainty modeling are crucial.***", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The method is evaluated on commonly used benchmarks.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The method demonstrates improved generalization on the benchmarks. However, the paper lacks sufficient architecture details for reproduction and full understanding. The authors should provide code to clarify these details. \n\nAdditionally, the method achieves only minor improvements on the ScanNet benchmark; the authors should discuss and analyze the reasons behind this.", "relation_to_broader_scientific_literature": "The method leverages a probabilistic framework, which may benefit other related tasks that require generalization capabilities.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agree to accept the papers. The author's rebuttal addressed reviewers' concerns, such as applications in other 3D representations like 3DGS. Please include all necessary changes in the final version.
Decision: Accept (poster)