Segment Anything in 3D with NeRFs
Authors: Jiazhong Cen
Zanwei Zhou
Jiemin Fang
chen yang
Wei Shen
Lingxi Xie
Dongsheng Jiang
XIAOPENG ZHANG
Qi Tian
Venue: NeurIPS 2023 poster (2023)
Keywords: Segmentation
NeRF
3D segmentation
TL;DR: We propose a generic and efficient methodology to lift the 2D Segment Anything Model to 3D with NeRFs, where only single-view prompts are required to segment the 3D object accurately within minutes.

Abstract: Recently, the Segment Anything Model (SAM) emerged as a powerful vision foundation model which is capable to segment anything in 2D images. This paper aims to generalize SAM to segment 3D objects. Rather than replicating the data acquisition and annotation procedure which is costly in 3D, we design an efficient solution, leveraging the Neural Radiance Field (NeRF) as a cheap and off-the-shelf prior that connects multi-view 2D images to the 3D space. We refer to the proposed solution as SA3D, for Segment Anything in 3D. It is only required to provide a manual segmentation prompt (e.g., rough points) for the target object in a single view, which is used to generate its 2D mask in this view with SAM. Next, SA3D alternately performs mask inverse rendering and cross-view self-prompting across various views to iteratively complete the 3D mask of the target object constructed with voxel grids. The former projects the 2D mask obtained by SAM in the current view onto 3D mask with guidance of the density distribution learned by the NeRF; The latter extracts reliable prompts automatically as the input to SAM from the NeRF-rendered 2D mask in another view. We show in experiments that SA3D adapts to various scenes and achieves 3D segmentation within minutes. Our research offers a generic and efficient methodology to lift a 2D vision foundation model to 3D, as long as the 2D model can steadily address promptable segmentation across multiple views.

Segment Anything in 3D with NeRFs
Jiazhong Cen't Zanwei Zhou' Jiemin Fang2.3 Chen Yang', Wei Shen Lingxi Xie?, Dongsheng Jiang', Xiaopeng Zhang', Qi Tian MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 2 Huawei Inc. 3 School of EIC, Huazhong University of Science and Technology wei. shen@sj tu. edu. cn
Abstract
Recently, the Segment Anything Model (SAM) emerged a powerful vision foundation model which is capable to segment anything 2D images. This paper aims to generalize SAM to segment 3D objects. Rather than replicating the data acquisition and annotation procedure which is costly in 3D, we design an efficient solution, leveraging the Neural Radiance Field (NeRF) as cheap and off- off-the- shelf prior that connects multi-view 2D images to the 3D space. We refer to the proposed solution as SA3D, for Segment Anything in 3D. It is only required to provide a manual segmentation prompt (e.g., rough points) for the target object single view, which is used to generate its 2D mask in this view with SAM. Next, SA3D alternately performs mask inverse rendering and cross-view self- prompting across various views to iteratively complete the 3D mask of the target object constructed with voxel grids. The former projects the 2D mask obtained by SAM in the current view onto 3D mask with guidance of the density distribution learned by the NeRF; The latter extracts reliable prompts automatically as the input to SAM from the NeRF-rendered 2D mask another view. We show in experiments that SA3D adapts to various scenes and achieves 3D segmentation within minutes. Our research reveals a potential methodology lift the ability of 2D vision foundation model to 3D, as long as the 2D model can steadily address promptable segmentation across multiple views. Our code is available /tte::///ttttbbb com/ caa.://ummatttmmmmtaaaaaaaaaaaaa
Introduction
The computer vision community has been pursuing a vision foundation model that can perform basic tasks (e.g., segmentation) in any scenario and for either 2D or 3D image data. Recently, the Segment Anything Model (SAM) emerged and attracted lot of attention, due to its ability to segment anything in 2D images, but generalizing the ability of SAM to 3D scenes remains mostly uncovered. One may choose to replicate the pipeline of SAM to collect and comi-automatically annotate large set of 3D scenes, but the costly burden seems unaffordable for most research groups.
We realize that an alternative and efficient solution lies in equipping the 2D foundation model (i.e., SAM) with 3D perception via 3D representation model. In other words, there is no need to establish 3D foundation model from scratch. However, there a prerequisite: the 3D representation model shall be capable to render 2D views and register 2D segmentation results to the 3D scene. Thus, we an off-the-shelf solution. NeRF family of algorithms that formulates each 3D scene into deep neural network that serves 3D prior connecting multiple 2D views.
use the Neural Radiance Fields (NeRF) 38 53
"Coreesponding author.
'Equal contribution.
PPooeect lead.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1 step
3rd step
Render view
Get the 3D object
NeRF
2dd step
Input prompts
Figure I: Given any pre-trained NeRF, SA3D takes prompts from one single rendered view as input and outputs the 3D segmentation result for the specific target.
As shown in Figure our solution is named Segment Anything in 3D (SA3D). Given NeRF trained on a set of 2D images, SA3D takes prompts (e.g., click points on the object) in a single rendered view input, which is used to generate 2D mask in this view with SAM. Next, SA3D alternately performs two steps across various views to iteratively complete the 3D mask of the object constructed with voxel grids. In each round, the first step is mask inverse rendering, in which the previous 2D segmentation mask obtained by SAM is projected onto the 3D mask via density-guided inverse rendering offered by the NeRF, The second step cross-view self-prompting. in which NeRF is used to render the 2D segmentation mask (which may be inaccurate) based on the 3D mask and the image from another view, then few point prompts are automatically generated from the rendered mask and fed into SAM to produce a more complete and accurate 2D mask. The above procedure is exccuted iteratively until all necessary views have been sampled.
We conduct various (e.g., object, part-level) segmentation tasks on the Replica [51 and NVOS 47 datasets. Without grraiiing/reddesigniin SAM or NeRF, SA3D easily and efficiently adapts to different scenarios. Compared to existing approaches, SA3D enjoys a simplified pipeline that typically completes 3D segmentation within minutes. SA3D not only offers an efficient tool for segmenting anything in 3D, but also reveals a generic methodology to lift 2D foundation models to the 3D space. The only prerequisite lies in the ability steadily address promptable segmentation across multiple views, and we hope it becomes a general property of 2D foundation models in the future.
2 Related Work
2D Segmentation Since FCN [36] has been proposed, research on 2D image segmentation has experienced a rapid growth. Various sub-fields of segmentation have been explored deeply by numerous studies 18 24 71. With transformers
T0 entering the field of segmentation, 63 have been proposed and the whole field recent significant breakthrough in this field is the Segment Anything Model (SAM) (25). As an emerging vision foundation model, SAM is recognized as potential game-changer, which aims to unify the 2D segmentation task by introducing a prompt- based segmentation paradigm. An analogous model to SAM is SEEM 75 which also exhibits impressive open- vocabulary segmentation capabilities.
[58
many new segmentation architectures [72] 7 6
52
of segmentation has been further developed.
3D Segmentation
Numerous methods have explored various types of 3D representations perform
3D segmentation. These scene representations include RGB-D images 660 62 64 &) point clouds 44 and grid space such voxels 21
45 770
55 35 cylinders 774 and bird's eye view space [67 16]
Although 3D segmentation has been developed for a period of time, compared with 2D segmentation, the scarcity of labeled data and high computational complexity make it difficult to design unified framework similar to SAM.
To tackle the limitation of data scarcity, many explored lifting 2D foundation models to 3D. In these studies, the most relevant work to SA3D is LERF [23), which trains a feature field of the Vision-Language Model (i.e,, CLIP (46) together with the radiance field. Compared with SA3D,
Lifting 2D Vision Foundation Models to 3D
previous studies 23 9 17
69
31
65
22

---

LERF focuses on coarsely localizing the specific objects with text prompts but not fine-graing 3D segmentation. The reliance on CLIP features makes it insensitive to the specific location information of the target object. When there are multiple objects with similar semantics in the scene, LERF cannot perform effective 3D segmentation. 'hhe remaining methods mainly focus on point clouds. By connecting the 3D point cloud with specific camera poses with 2D multi-view images, the extracted features by 2D foundation models can be projected to the 3D point cloud. The data acquisition of these methods is more expensive than ours, í.c., acquiring multi-view images for NeRFs.
Segmentation in NeRFs Neural Radiance Fields (NeRFs) [8 53
40
19
B3
61
30
12
are a series of 3D implicit representation. Inspired by their success 3D consistent novel view synthesis, numerous studies have delved into the realm of 3D segmentation within NeRFs. Zhi el al. 73 proposes Semantic-NeRF, a method that incorporates semantics into appearance and geometry. They showcase the potential of NeRFs in label propagation and refinement. NVOS 47 introduces an interactive approach to select 3D objects from NeRFs by training a lightweight multi- layer perception (MLP) using mnstom-designe
3D features. Other approaches. e.g. N3F [57). DFF [27]. LERF [23] and ISRF [15]. aim to lift 2D visual features to 3D through training additional feature fields. These methods are required re-design/-train NeRF models and usually involve additional feature-matching processes. There are also some other instance segmentation and semantic segmentation approaches [50
68 34 20
14 59 28 combined with NeRFs.
The most closely related approach to our SA3D is MVSeg 39 a component of SPIn-NeRF [39], which focuses on NeRF inpainting. MVSeg adopts video segmentation techniques to propagate a 2D) mask across different views and employs these masks as labels for training Semantic-NeRF model. However, video segmentation models lack explicit 3D) structure information, which hard to handle significant occlusions in complex scenes. Our method aims at building NeRF-driven consistency across views based on ooffprrompinng and lifting 2D masks robust 3D masks.
3 Method
In this section, we first give a brief review of Neural Radiance Fields (NeRFs) and the Segment Anything Model (SAM). Then we introduce the overall pipeline of SA3D, Finally, we demonstrate the design of each component in SA3D in detail.
3.1 Preliminaries
Neural Radiance Fields (NeRFs) Given training dataset I of multi-view 2D images, NeRFs [8 learn a function fo (x, d) - a), which maps the spatial coordinates x R and the view direction d g of point into the corresponding color R3 and volume density a R. 0 denotes the learnable parameters of the function f which usually represented by a multi-layer perceptron (MLP). To render image Io, each pixel undergoes ray casting process where ray r(t) = x, + td is projected through the camera pose. Here, x, is the camera origin, d is the ray direction, and f denotes the distance of a point along the ray from the origin. The RGB color (r) at the location determined by ray is obtained via a differentiable volume rendering algorithm:
To(r)= 1"² ((r(t))c(((t)dd) d)dt,
(1)
where w(r(t)) = exp(- H (r(ss)dss) (r(()), and f, and ły denote the near and far bounds of the ray, respectively.
Segment Anything Model (SAM) SAM 25 takes an image I and of prompts P as input, and outputs the corresponding 2D segmentation mask MS in the form of a bitmap, i.e.,
M s(I,
(2)
The prompts p P can be points, boxes, texts, and masks.
3.2 Overall Pipeline
We assume that we already have a NeRF model trained on the dataset I. Throughout this paper, unless otherwise specified, opt to employ the TensoRF 3 the NeRF model, considering its
3

---

Mask
M
1 view
Image Rendering
Input Prompt
Mask
Inverse Rendering
Mask
@
Rendering
M
Ma
2 view
Cross-view
Mask
Completing
Self-prompting
Inverse Rendering
Nett
e
Mask
1) view
Rendering
M
M
Cross-view
Self-prompting
Inverse Rendering
Figure 2: The overall pipeline of SA3D. Given a NeRF trained on a set of multi-view 2D images, SA3D first takes prompts in single view for the target object as input and uses SAM produce 2D mask in this view with these prompts. Then, SA3D performs an alternated process of mask inverse rendering and cross-view self-prompiing complete the 3D mask of the target object constructed with voxel grids. Mask inverse rendering is performed to project the 2D mask obtained by SAM onto the 3D mask according to the learned density distribution embedded in the NeRF. Cross-vier self-prompting is conducted to extract reliable prompts automatically as the imput to SAM from the NeRF-rendered 2D mask given a novel view. This alternated process executed iteratively until we get the complete 3D mask.
superior efficiency of training and rendering. As shown in Figure an image I from a specific view is first rendered with the pre-trained NeRF model. A set of prompts (e.g., in this paper, we often use set of points), pin, is introduced and fed into SAM along with the rendered image. The segmentation mask M of the according view is obtained, which is then projected onto the 3D mask VE E 3 constructed voxel grids with the proposed mask inverse rendering technique (Section 3.3). Then 2D segmentation mask (()) from novel view is rendered from the 3D mask. The rendered mask is usually inaccurate. We propose a cross-view self-prompting method (Section 3.4) to extract point prompts p(n) from the rendered mask and further feed them into SAM. Thus more accurate 2D mask M in this novel view is produced and also projected onto the voxel grids to complete the 3D mask. The above procedure is executed iteratively with more views traversed. Meanwhile, the 3D mask become more and more complete. The whole process bridges 2D segmentation results with 3D ones efficiently. Noting that no neural network needs to be optimized except the 3D mask.
3.3 Mask Inverse Rendering
As shown in Equation (I) the color of each pixel in a rendered image is determined by sum of weighted colors along the corresponding ray. The weight (r())) reveals the object structure within the 3D space, where high weight indicates the corresponding point close the object's 's surface. Mask inverse rendering aims to project the 2D mask to the 3D space to form the 3D mask based on these weights.
Formally, the 3D mask represented as voxel grids V e xxxx where each grid vertex stores inro-initialized soft mask confidence score. Based on these voxels grids, each pixel of the 2D mask from one view is rendered as
M(r) = 1.' ((((())
where r(t) is the ray casting through the mask pixel, (r(()) is inherited from density values of the pre-trained NeRF, and V(r(t) denotes the mask confidence score at the location r(t) obtained from voxel grids v Denote M (r) as the corresponding mask generated by SAM. When M (r)
(r(()) is computed by trilinearly interpolating vertex values of the 3D mask grids.

---

the goal of mask inverse rendering is to increase V(r(t) with respect to (r(())) In practice, this can be optimized using the gradient descent algorithm. For this purpose, we define the mask projection loss as the negative product between Mra and M(r):
Cpros
Σ PER(T)
MaAn(T) (M()),,())),
where R(I) denotes the ray set of the image I.
The mask projection loss is constructed based on the assumption that both the geometry from the NeRF and the segmentation results of SAM are accurate. However, in practice, this is not always the case. We append a negative refinement term the loss optimize the 3D mask grids according to multi-view mask consistency:
Cpros =
Σ PE PE)()
Msa((r) M(r) + Σ (1-Masn) M(r),
(5)
FER(I)
where À is hyper-parameter to determine the magnitude of the negative term. With this negative refinement term, only if SAM consistently predicts region as foreground from different views, SA3D marks its corresponding 3D region as foreground. In each iteration, the 3D mask V is updated &Cyuz with gradient descent, where denotes the learning rate.
via V V
17 BV
3.4 Cross-view Self-prompting
Mask inverse rendering enables projecting 2D masks into the 3D space form the 3D mask of target object. To construct accurate 3D mask, substantial 2D masks from various views need to be projected. SAM can provide high-quality segmentation results given proper prompts. However, manually selecting prompts from every view is time-consuming and impractical. We propose a cross-view prof-proppinng mechanism to produce prompts for different novel views automatically.
Specifically, we first render novel-view 2D segmentation mask (()) from the 3D mask grids V according to Equation (3). This mask is usually inaccurate, especially at the preliminary iteration of SA3D. Then obtain some point prompts from the rendered mask with specific strategy. The above process named cross-view self prommping.. While there are multiple possible solutions for this strategy, present feasible that has been demonstrated to be effective.
Self- promppromp Strategy Given an inaccurate 2D rendered mask M("), the self-promp strategy aims extract set of prompt points P, from it, which can help SAM to generate 2D segmentation result accurate as possible. It is important to note that ( is not typical 2D bitmap, but rather confidence score map computed using Equation (3). Since each image pixel p corresponds to ray r in rendered view, we use p for an easier demonstration of the prompt selection strategy images.
As P. is initialized to an empty set, the first prompt point Po selected as the point with the highest mask confidence score: Po arg maxp ( (p). To select new prompt points, first mask out square shaped region& (n)) centered with each existing point prompt P Pa Considering the depth 2() can be estimated by the pre-trained NeRF, we transform the 2D pixel p to the 3D point ((p) (x((())), v(G(()),,(()())))
(g(( (P) ( P <(p) (g (P)
(P x(p)
(6)
where z(p). y(p) denote the 2D coordinates of p. and K denotes the camera intrinsics. The new prompt point is expected to have a high confidence score while being close to existing prompt points. Considering the two factors, we introduce decay min-max normalized Euclidean distance. For each remaining point p in M(n), the decay term is
the confidence score. Let d(::-) denote the
(* (p) = min(M( (p) ((((p).(())) p E P.}.
Then decayed mask confidence score M (p) is computed as
M (p) M (p) AM
(8)
TTh side length of the region is the radius of circle whose equal the rendered mask (n).
5

---

(IoU) between the rendered mask (n)) and the SAM prediction M If the IoU falls below predefined threshold т, indicates a poor overlap between the two masks. The prediction from SAM Figure 3: Some visualization results in different scenes (LERF-donuts 23 LERF-figurines, Replica- room0 (51) and 360-kitchen I).
The remaining point with the highest decayed score, i.e., p" arg maXpgp, ( (p). is added to the prompt P. P. U {p"}. The above selection process is repeated until either the number of prompts [P&l reaches a predefined threshold ny or the maximum value ( (p) is smaller than
IoU-aware View Rejection When the target object is rendered in heavily occluded views, SAM may produce incorrect segmentation results and degrades the quality of the 3D mask. To avoid such situations, we introduce an additional view rejection mechanism based on the intersection-over
is rejected, and the mask inverse rendering step is skipped in this iteration.
4 Experiments
this section, we quantitatively evaluate the segmentation ability of SA3D on various datasets. Then, qualitatively demonstrate the versatility of SA3D, which can conduct instance segmentation, part segmentation, and text-prompted segmentation etc.
4.1 Datasets
For quantitative experiments, we use the Neural Volumetric Object Selection (NVOS) 47 SPIn- NeRF (39), and Replica (5T) datasets. The NVOS [47] dataset based on the L.FFF dataset [37], which includes several forward-facing scenes. For each scene, NVOS provides reference view

---

with scribbles and a target view with 2D segmentation masks annotated. Similar to NVOS, SPIn- NeRF [39 annotates some data manually to evaluate interactive 3D segmentation performance. These
annotations are based on some widely-used NeRF datasets [37 38 26 13 The Replica [51] dataset provides high-quality reconstruction ground truths of various indoor scenes, including clean dense geometry. high-resolution and high-dynamic-range
textures, glass and mirror surface infor- mation, semantic classes, planar segmentation, and instance segmentation masks. For qualitative analysis, we use the LLFF [37] dataset and the 360 dataset [1]. SA3D is further applied to the LERF 23 dataset, which contains more realistic and challenging scenes.
4.2 Quantitative Results
NVOS Dataset For fair comparisons, we fol- low the experimental setting of the original NVOS 47). We first scribble on the reference view (provided by the NVOS dataset) to conduct 3D segmentation, and then render the 3D seg- mentation result the target view and evaluate the loU and pixel-wise accuracy with the pro- vided ground truth. Note that the scribbles are preprocessed to meet the requirements of SAM. More details can be found in the supplement. As shown in Table [ SA3D outperforms previous approaches by large margins, i.., +6.5 mloU over the previous SOTA ISRF and +20.2 mIoU over the NVOS.
Table I: Quantitative results on NVOS.
Method
mloU (%)
mAcc (%)
Graph-cut (3D) 48 47
39.4
73.6
NVOS 47
70.1
92.0
ISRF [IS]
83.8
96.4
SA3D (ours)
90.3
98.2
Table 2: Quantitative results the SPIn-NeRL dataset.
Single view
MVSeg 39
SA3D (ours)
Scenes
loU (%)
Ace (%)
IoU (%)
Acc (%)
loU (%)
Acc (%)
Orchids
79.4
96.0
92.7
98.8
83.6
96.9
Leaves
78.7
98.6
94.9
99.7
97.2
99.9
Fern
95.2
99.3
94.3
99.2
97.1
99.6
Room
73.4
96.5
95.6
99.4
88.2
98.3
Horns
85.3
97.1
92.8
98.7
94.5
99.0
Fortress
94.1
99.1
97.7
99.7
98.3
99.8
Fork
69,4
98.5
87.9
99.5
89.4
99.6
Pinecone
57.0
92.5
93.4
99.2
92.9
99.1
Truck
37.9
77.9
85.2
95.1
90.8
96.7
Lego
76.0
99.1
74.9
99.2
92.2
99.8
74.6
95.5
90.9
98.9
92.4
98.9
SPIn-NeRF Dataset We follow SPIn-NeRF 39 conduct label propagation for evaluation. Given specific reference view of target object, the 2D ground-trute truth mask of this view is available. The prompt input operation is omitted, while the 2D ground-trutt mask of the target object from the reference view is directly used for the initialization of the 3D mask grids. This is reasonable since in most situations users can refine their input prompts to help SAM generate 2D mask as accurately as possible from the reference view. Once the 3D mask grids are initialized, the subsequent steps are exactly the same as described in Section With the 3D mask grids finalized, the 2D masks in other views are rendered to calculate the IoU with the 2D ground-trutt masks. Results can be found in Table SA3D is demonstrated to be superior in both forward -fcing and 360 scenes.
"Single view" refers to conducting mask inverse rendering exclusively for the 2D ground-truu mask of the reference view. This process equivalent mapping the 2D mask the 3D space based on the corresponding depth information, without any subsequent learnable/updating step. We present these results to demonstrate the gain the alternated process in our framework. As shown in Table 2 SA3D outperforms MVSeg 39 in most scenes, especially +5.6 mloU on Truck and + +17.3 mloU on Lego. Besides, compared with the "Single view" model, significant promotion is achieved, i.e. +17.8 mloU, which further proves the effectiveness of our method.
In Tables 2 and 3
7

---

Table 3: Quantitative results on Replica (mloU).
Scenes
office0
officel
office2
office3
office4
room0
room!
room2
mean
Single view
68.7
56.5
68.4
62.2
57.0
53.4
53.8
56.7
59.8
MVSeg (39
31.4
40.4
30.4
30.5
25.4
31.1
40.7
29.2
32.4
SA3D (ours)
84.4
77.0
88.9
84.4
82.6
77.6
79.8
89.2
83.0
Text: "The table"
Text: "The vasc"
Text:
"The bonsai"
Figure 4: 3D segmentation results of SA3D with the text prompts in 360-garden [I]
Replica Dataset We use the processed Replica data with 2D instance labels provided by Zhi al. [73] to evaluate the segmentation performance of SA3D. We retrieve all views containing each reference view. With similar setting of experiments dataset, we use the ground-truth mask of the reference view and perform SA3D to conduct label propagation for evaluation. For each scene in Replica, around 20 objects
object and specify
the SPIn-NeRF
chosen for evaluation. Refer to the supplement for more details. As shown in Table 3 the mean IoU (mloU) is reported for all available objects in different scenes. We exclude the pixel-wise accuracy metric since an object only appears in a few views in the indoor scenes of Replica, where the pixel-wise accuracy is too high to serve a reliable metric.
In complex indoor scenes of Replica, MVSeg's 's strategy based on video segmentation proves to be inef- fective, which generates numerous inaccurate 2D pseudo-labels, for refinement, Consequently, the final 3D segmentation results of MVSeg even underperform those achieved by the "Single view" method. In contrast, SA3D accurately captures segmented objects in complex 3D scenes. Visualization results are shown in Figure 3
using the Semantic-NoRF [73]
4.3 Qualitative Results
We conduct three kinds of segmentation tasks: object segmentation, part segmentation and text- prompting segmentation. The first two are the core functions of SA3D. As shown in Figure 3 SA3D demonstrates its capability to segment diverse 3D objects across different scenes, even when the objects are of small scales. Besides, SA3D can also handle challenging part segmentation. The last row the figure showcases SA3D's precise segmentation of the bucket, small wheel, and dome light of the Lego bulldozer. Figure 4 demonstrates the potential of SA3D in combining with language models. Given a text phrase, the corresponding object can be accurately cut out. The text text-prompting segmentation is built upon Grounding-DINO (33) model capable of generating bounding boxes for objects based on text prompts. These bounding boxes serve as input prompts for SA3D in the segmentation process.
Table 4: Ablation on different numbers of views for 3D mask generation. Numbers in parentheses represent the view percentage of total training views.
Number of Views
(10%)
(20%)
21 (50%)
43 (100%)
IoU on Fortress (forward facing)
97.8
98.3
98.3
98.3
Time Cost (s)
7.6
12.8
29.0
59.0
Number of Views
11 (10%)
21 (20%)
51 (50%)
103 (100%)
IoU on Lego (3609)
84.5
84.8
91.5
92.2
Time Cost (s)
23.5
43.5
103.8
204.9

---

4.4 Ablation Study
Number of Views The process of mask inverse rendering and cross-view view self- ceom-proptinggggg is alter- nated across different views. By default, we utilize all available views in the training set I. However, to expedite the 3D segmentation procedure, the number of views can be reduced. As shown in 4 We perform experiments on two representative scenes from the SPIn-NeRF 39 dataset to demonstrate this characteristic. The views are uniformly sampled from the sorted training set. In forward facing scenes where the range of the camera poses is limited, satisfactory results can be achieved by selecting only a few views. On an Nvidia RTX 3090 GPU, the 3D segmentation process with 5 views can be completed within 10 seconds. On the contrary, in scenes where the range of camera poses is wider, larger number of views are required to yield greater improvements. Note that even with 50 views, the segmentation task can still be completed in less than 2 minutes.
Hyper-parameters SA3D involves three hyper-parameters: the IoU rejection threshold т, the loss balance coefficient À in Equation ( and the number of self-prompting points npp As shown in Table too small 7 values lead to unstable SAM predictions, introducing noises to the 3D mask; too large values impede the 3D mask from getting substantial information. Table 7 indicates slightly introducing a negative term with the factor can reduce noise for mask projection. However, too-large negative term may make the mask completion process unstable and causes degraded performance. The selection of np depends on the specific segmentation target. as SAM tends to produce over-segmented results that capture finer details of objects. As shown in Figure [] for objects with relatively large scale and complex structures, a bigger ?p produces better results. Empirically. setting np 3 can meet the requirements of most situations.
Self- Self-prompting Strategy Without the 3D distance based confidence decay (Equation our self- self-prompting strategy degrades to simple 2D NMS (Non-Maximum Suppression), which selects prompt point with the highest confidence score and then masks out a region around it. To showcase the efficacy of our design, conduct experiments using the NVOS benchmark and presented per-scene results for in-depth analysis.
Table 5: Ablation on the confidence decay term of the self- self-prompting strategy.
w/ Confidence Decay Term
w/o Confidence Decay Term
Scenes
IoU (%)
Acc (%)
IoU (%)
Acc (%)
Fern
82.9
94.4
82.9
94.4
Flower
94.6
98.7
94.6
99.7
Fortress
98.3
99.7
98.4
99.7
Horns (center)
96.2
99.3
96.2
99.3
Horns (Left)
90.2
99.4
88.8
99.3
Leaves
93.2
99.6
93.2
99.6
Orchids
85.5
97.3
85.4
97.3
Trex
82.0
97.4
64.0
93.3
mean
90.3
98.2
87.9
97.7
Table 5 shows that simple NMS self- self-prompting is enough for most cases. But for hard cases like "LLFF-trex' (a trex skeleton, as shown in Figure where a large number of depth jumps, the confidence decay term contributes lot. In such situation, inaccurate masks bleed through gaps in the foreground onto the background. If the self-prompting mechanism generates prompts on these inaccurate regions, SAM may produce plausible segmentation results that cheat the loU-rejection mechanism and finally the segmentation results will involve unwanted background regions.
2D Segmentation Models In addition SAM, also incorporate four other prompt-based 2D segmentation models 75 49 32 5 into our framework to demonstrate the generalization ability of SA3D. The evaluation results the NVOS dataset is shown in Table
Discussion
top of the experimental results, we hope to deliver some insights from our preliminary study of integrating SAM and NeRF, i.e., 2D foundation model and 3D representation model.

---

Table 6: Ablation on different loU-aware rejection threshold r on the Replica office0.
Table 7: Ablation on different negative term coef- ficient on the Replica office0.
T
0.3
0.4
0.5
0.6
0.7
À
0.05
0.1
0.15
0.3
0.5
mloU
74.9
79.7
84.4
81.3
82.0
mloU
79.1
82.9
84.4
84.9
83.3
Table Ablation on different 2D segmentation models.
SEEM
75
SimpleClick (32
RITM 49
FocalClick [
mIoU (%)
mAce (%)
mIoU (%)
mAcc (%)
mloU (%)
mAce (%)
mloU (%)
mAcc (%)
86.0
97.0
87.7
97.8
81.2
96.3
88.9
98.1
Input image prompts
sclf-prompting points
sclf-prompting points
10 self-prompting points
Figure 5: Results of different self-prompting points numbers np the LLFF-trex 37
scene.
SAM prediction
SA3D prediction
Figure The 2D segmentation result by SAM and the 3D segmentation result by SA3D of the LLFF-fem 37 scene. SA3D produces more details that are missing in the SAM segmentation result.
First, NeRF improves the segmentation quality of SAM. In Figure 6 we show that SA3D can eliminate segmentation errors of SAM and effectively capture details such as holes and edges. Perceptually, SAM, as well as other 2D perception models, is often sensitive to the viewpoint, and NeRF offers the ability of 3D modeling and hence complementariness inspires that using NeRF or other 3D structural priors is a resource- efsuucceefffccentt method to lift a vision foundation model from 2D to 3D, as long as the foundation model has ability to self-prompt. This methodology can save many resources because collecting large corpus of 3D data is often costly. We look forward to research efforts to enhance the 3D perception ability of 2D foundation models (e.g., injecting 3D-aware loss into 2D pre-training).
assist recognition. Additionally, SA3D
Limitation SA3D has limitations in panoptic segmentation. First, the current paradigm relies on the first-view prompt. If some objects do not appear in the view for prompting, they will be omitted the subsequent segmentation process. Second, the same part in the scene may be segmented into different instances with similar semantics in different views. This ambiguity cannot easily eliminated under the current mechanism design and lead to unstable training. We leave these issues as future work.
6 Conclusion
In this paper, we propose SA3D, a novel framework that generalizes SAM segment 3D objects with neural radiance fields (NeRFs) as the structural prior. Based on trained NeRF and set of prompts a single view, SA3D performs an iterative procedure that involves rendering novel views, self- ceff-prompting SAM for 2D segmentation, and projecting the segmentation back onto 3D mask grids. SA3D can efficiently applied to wide range segmentation tasks. Our research sheds light resource-efficient methodology that lifts vision foundation models from 2D to 3D.
10

---

Acknowledgement
This work was supported by NSFC 62322604, NSFC 62176159, Natural Science Foundation of Shanghai 21 and Shanghai Municipal Science and Technology Major Project 2021SHZDZX012.. Specifically, We extend our sincere thanks to Weichao Qiu for his insight- ful suggestions during the course of this work.
References
[1] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[2] Wang Bing, Lu Chen, and Bo Yang. Dm-nerf: 3d scene geometry decomposition and manipula- tion from 2d images, arXiv preprint arXiv:2208.071 07227, 2022.
[3] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
[4] Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell., 2018.
[5] Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. Fo- calclick: Towards practical interactive image segmentation. In CVPR, 2022.
[6] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
[7] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021.
[8] Hang Chu, Wei-Chiu Ma, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler. Surfconv: Bridging 3d and 2d convolution for rgbd images. CVPR, 2018.
[9] Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR, 2023.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[11] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation complex scenes. arXiv preprint arXiv:2209.08777 2022.
[12] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nieβner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIG- GRAPH Asia 2022 Conference Papers, 2022.
[13] Sara Fridovich- Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[14] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, An- dreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In 3DV, 2022.
[15] Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and PJ Narayanan. Interactive segmentation of radiance fields. arXiv preprint 22. 2022.
[16] Nikhil Gosala and Abhinav Valada. Beee'v-eee- panoptic segmentation using monocular frontal view images. IEEE Robot. Autom. Lett., 2022.

---

[17] Huy Ha and Shuran Song. Semantic abstraction: Open- 3d scene understanding from 2d vision- vision-language models. In CoRL, 2022.
[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, 2017.
[19] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul E. Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV, 2021.
[20] Benran Hu, Junkai Huang, Yichen Liu, Yu-Wing Tai, and Chi-Keung Tang. Instance neural radiance field. arXiv preprint urXiv:2304.04395, 2023.
[21] Jing Huang and Suya You. Point cloud labeling using 3d convolutional neural network. In ICPR, 2016.
[22] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer. Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241, 2023.
[23] Justin Kerr, Chung Min Kim. Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553, 2023.
[24] Alexander Kirillov, Kaiming He, Ross B. Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019.
[25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304. 02643, 2023.
[26] Arno Knapitsch, Jaesik Park, Qian Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Trans. Graph., 2017.
[27] Sosuke Kobayashi, Eicchi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In NeurlPS, 2022.
[28] Ruofan Liang, Jiahao Zhang. Haoda Li, Chen Yang, and Nandita Vijaykumar. Spidr: Sdf-based neural point fields for illumination and deformation. arXiv preprint arXiv:2210.08398, 2022.
[29] Yen-Chen Lin, Pete Florence, Jonathan T. Barron, Tsung-Y Yi Lin, Alberto Rodriguez, and Phillip Isola. Nerf-supervision: Learning dense object descriptors from neural radiance fields. In ICRA, 2022.
[30] David B. Lindell, Julien N. P. Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In CVPR, 2021.
[31] Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su. Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models. In CVPR. 2023.
[32] Qin Liu, Zhenlin Xu, Gedas Bertasius, and Mare Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. ICCV, 2023.
[33] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.cm 05499, 2023.
[34] Xinhang Liu, Jiaben Chen, Huai Yu, Yu-Wing Tai, and Chi-Keung Tang. Unsupervised multi- view object segmentation using radiance field propagation. In NeurIPS, 2022.
[35] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. NeurIPS, 2019.
[36] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
12

---

[37] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi. Ren Ng, and Abhishek Kar. Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph., 2019.
[38] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[39] Ashkan Mirzaei, Tristan Aumentado-Armstrony Konstantinos G. Derpanis, Jonathan Kelly, Marcus A. Brubaker, Igor Gilitschenski, and Alex Levinshtein. SPIn-NeRF: Multiview segmen- tation and perceptual inpainting with neural radiance fields. In CVPR, 2023.
[40] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph., 2022.
[41] Michael Niemeyer and Andreas Geiger. GIRAFFE: representing scenes as compositional
generative neural feature fields. In CVPR, 2021.
[42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
[43] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023.
[44] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[45] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. NeurIPS, 2017.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[47] Zhongzheng Ren, Aseem Agarwala, Bryan C. Russell, Alexander G. Schwing, and Oliver Wang. Neural volumetric object selection. In CVPR, 2022.
[48] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut": interactive foreground extraction using iterated graph cuts. ACM Trans. Graph., 2004.
[49] Konstantin Sofiiuk, Ilya A Petrov, and Anton Konushin. Reviving iterative training with mask guidance for interactive segmentation. In ICIP, 2022.
[50] Karl Stelzner, Kristian Kersting, and Adam Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01 2021
[51] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint 1w0.iv::006.5000
2019.
[52] Robin Strudel, Ricardo Garcia Pinel, Ivan Laptev, and Cordelia Schmid. Segmenter: Trans- former for semantic segmentation. In ICCV, 2021.
[53] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.
[54] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprint arXiv:2212.13545. 2022.
[55] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse voxet-voxel convolution. In ECCV, 2020.
13

---

[56] Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng. Delicate textured mesh recovery from nerf via adaptive surface refinement. arXiv preprint arXiv:2303.0209) 2023.
[57] Vadim Tschemezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. In 3DV, 2022.
[58) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[59] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. arXiv preprint arXi::2111.13200, 2021.
[60] Weiyue Wang and UIrich Neumann. Depth-aware enn for rgb-d segmentation. In ECCV, 2018.
[61] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwa- janakorn. Nex: Real-time view synthesis with neural basis expansion. In CVPR, 2021.
[62] Zongwei Wu, Guillaume Allibert, Christophe Stolz, Chao Ma, and Cédric Demonceaux. Depth- adapted cnns for rgb-d semantic segmentation. arXiv preprint arXiv:2206.03392 2022.
[63] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, 2021.
[64] Yajie Xing, Jingbo Wang, and Gang Zeng. Mallcable 2.5 d convolution: Learning receptive fields along the depth-axis for rgb-d scene parsing. In ECCV. 2020.
[65] Jihan Yang, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Regionple: Regional point-language contrastive learning for open-world 3d scene understanding, arXiv preprint orXiv:2304.0096 2023.
[66] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T Barron, and Ben Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv preprint arXiv:2302. /4859, 2023.
[67] Dongqiangzi Ye, Zixiang Zhou, Weijia Chen, Yufei Xie, Yu Wang, Panqu Wang, and Hassan Foroosh. Lidarmultinet: Towards unified multi-task network for lidar perception. arXiv preprint arXiv:2209.099 09385, 2022.
[68] Hong-Xing Xing Yu, Leonidas J. Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In ICLR, 2022.
[69] Junbo Zhang, Rumpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open- world 3d scene representations from 2d dense clip, arXiv preprint arXiv:2303.0477 04748, 2023.
[70] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.
[71] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR, 2017.
[72] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yan- wei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from a sqquence-to-sequeee
perspective with transformers. In CVPR, 2021.
[73] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger. and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV, 2021.
[74] Hui Zhou, Xinge Zhu, Xiao Song, Yuexin Ma, Zhe Wang, Hongsheng Li, and Dahua Lin. Cylinder3d: An effective 3d framework for driving-scene lidar semantic segmentation. arXiv preprint arxiv:2008. 51500, 2020.
[75] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.0677 2023.
14

---

Appendix
The appendix includes the following contents:
Implementation details (Section A
2. A segmentation refinement strategy (Section B
3. The scribble to point strategy for the evaluation of NVOS (Section C
4. An analysis about vanilla NeRF [8 used in SA3D (Section D):
5. More information about the object filtering of Replica [1 (Section
6. A further illustration of the self seomppron strategy (Section[p
7. An analysis about the effect of different kinds of occlusion NeRF (Section G
8. More visualization results with different kinds of input prompts (Section H
A
Implementation Details
We implement SA3D using Py 42 with reference to the code provided by DVGOv2 [54]. The SA3D model is built and trained on single Nvidia Geforce RTX3090 GPU. For our NeRF model, we primarily employ TensoRF [3) utilizing the VM-48 representation to store the radiance latent vectors. The radiance fields pre-trained for most datasets with 40,000 iterations. For the LLFF dataset (7] and the 360 dataset [1] the radiance fields are trained with 20,000 iterations.
B Refinement with A Two-pass Segmentation Mechanism
SAM may produce segmentation masks containing undesired parts. The IoU-aware view rejection is hard to handle this issue when the mis-classified region gradually expands.
We propose a two-pass segmentation mechanism to further refine the segmentation result. After completing 3D segmentation introduced the main manuscript, get a 3D mask V. To detect the mlas-slassffid region from V, we re-render the 2D segmentation mask M of the user-specific reference view and compare with the original SAM segmentation result M
Subsequently, we reset the original 3D mask to be a zero tensor and introduce another 3D mask V IR3 that specifically indicates the mis-classified regions. The 3D segmentation process is then repeated, with the key difference being the incorporation of negative prompt points during the self-prompting phase. In other words, the prompts obtained from V' serve as negative prompts for V. and vice versa. This incorporation of negative prompts enables SAM to gain a better understanding of the user's requirements and refine the segmentation accordingly (shown in Figure [B7. is important to note that while this two-pas pa-sasss segmentation mechanism holds promise, it was not utilized in our main experiments due to considerations of efficiency.
Vanilla SA3D
Two-pass Segmentation
Figure B7: The effect of the two-pass segmentation refinement.
C The Scribble to Points Strategy for The Evaluation of NVOS
The NVOS dataset provides a reference view and the corresponding scribbles for each scene (shown in Figure C8 In practice, since the scribbles usually contain tens of thousands of dense
15

---

Figure C8: Some scribbles on the reference views provided by NVOS and the corresponding segmentation results of SA3D. Blue scribbles are positive and red scribbles are negative.
points, SAM 25 cannot directy take such scribbles as input. The abundance of points the scribbles hinders SAM's performance when directly utilizing them as prompts, which is inherent limitation of SAM.
For fair comparison, extract positive and negative prompt points from the provided positive and negative scribbles, respectively. For input seribbles, we first skeletonize them and then select 2% points from the skeletonized positive scribbles the positive prompts and 0.5% points from the skeletonized negative scribbles as the negative prompts.
D
The Effect of Different NeRFs Used in SA3D
adapt SA3D to vanilla NeRF 33 to showcase generalizability. We present visualization results on the LLFF dataset. As illustrated in Figure D9 SA3D with the vanilla NeRF exhibits excellent performance without the need for additional modifications.
E
Object Filtering for The Replica Dataset
The Replica dataset contains many objects in each scene. However, it is important note that many of these objects exhibit low quality, as depicted in Figure making them unsuitable for evaluating 3D segmentation. Generally, these instances exhibit the following issues: some instances are not present in the training frames provided by Zhi et al. [73]; some instances are too small to be effectively segmented, such as thin slits doors; and some instances consist of unrecognizable, low-resolution pixels, such as blurred tags, which are not suitable for accurate instance segmentation. Accordingly, we carefully select approximately 20 representative instances from each scene for the evaluation. The list of instance IDs for each the quantitative results without object filtering in Table E9 Even without object filtering, SA3D demonstrates improvements compared to the single-view baseline.
be found in Table We have also included
Table E9: Quantitative results on Replica (mloU) without object filtering.
Scenes
office0
officel
office2
office3
office4
room0
room1
room2
mean
Single view
58.8
53.1
61.2
54.2
56.7
51.0
58.6
58.3
56.49
SA3D (ours)
65.1
59.8
69.7
61.4
63.8
56.8
68.4
72.2
64.65
An Illustration for The Proposed Self-prompting Strategy
We offer an illustration (Figure FI1 self- self-prompting strategy.
to assist readers in gaining a clearer understanding of the
16

---

Flower
Fortress
Fern
Orchids
Trex
Figure D9: 3D Segmentation results based on the vanilla NeRF on the LLFF dataset.
17

---

Figure E10: Some ground truth masks (shown in green) and their corresponding instances from the Replica dataset. These unreasonable segmentation targets are filtered out in evaluation.
Table E10: The selected id lists of Replica.
Scenes
ID list
office0
officel
office2
01114658882222233333344556666683 101,111
office3
office4
room0
rooml
12443990011160144422222644444444
room2
35577381112200122226777333333899
In the self- self-prompting strategy, prompt points Pa are derived from an incomplete 2D rendered mask M(n), which represented confidence score map. Initially, the selected prompt points set P. empty, and the first prompt point Po is selected as the with the highest confidence score in the mask M(n), For subsequent points, square regions centered around existing prompt points are masked out on (()), The depth =p), estimated by the pre-trained NeRF, helps convert 2D pixel P into a 3D point (p)). The new prompt point expected to have a high confidence score while being close to existing prompt points. Hence, a distance-ave aware decay term is introduced to compute the confidence score. The remaining point with the highest decayed score is added the prompt set. This selection process is repeated until either the number of prompts [P, reaches a predefined threshold np or the maximum value of the remaining points is less than O. Please refer to Section 3.4 of the main manuscript for more details.
G The Effect of Occlusion in NeRF
There are two cases of occlusion in NeRFs: part of the target object does not appear in some views (but appear in other views), or it does not appear at all. For the former case, SA3D can recover it using information from other views; for the latter case, there can be parts missing in the target object (see Figure G12). This is interesting future direction (e.g., applying generative models such as diffusion models).
H More Visualization Results
We present additional visualization results in Figure HT3 and Figure HI5
showcasing the effectiveness
of SA3D across various input prompts. We also provide some visualization of extracted meshes of the segmented objects (Figure H14) show the extracted 3D geometry. Please note that the quality of these meshes can be further improved by applying more effective NeRF2Mesh methods [56] 66
18

---

3D distance-based confidence decay Equation (6),(7)(8)
Mask square region
Select point with the highest confidence
Mask squire regions
Figure FI1: An illustration of the self-prompting strategy.
eizchded but seep in other views
evver seen in any vis
Figure G12: The effect different kinds occlusion in NeRF.
"The sheep"
apple"
"The cup"
"The bug"
"The dish"
paper"
Figure H13; Text prompt based visualization results on the LERF figurines dataset.
Figure H14: Some visualization results of the extracted meshes of the segmented objects.
19

---

Forward facing
BEER
Chalkhoard
Fern
Sheerack
360
Kitchen
Bonaal
Figure H15: More visualization results the LLFF dataset and the 360 dataset (based on point and box prompts).
20


Reviews:
[{"review_id": "qQXhLXvgR6", "reviewer": "Reviewer_Xgwo", "summary": "The authors propose a combination of the newly introduced Segment-Anything Model (SAM) with Nerf, yielding the  Segment Anything in 3D (SA3D) system.\nSA3D cam take a 3D scene reconstructed by Nerf and based on a user prompt (e.g. a few keypoints) can carve out distinct 3D objects from the scene. This is shown to outperform previous state-of-the-art systems. \nThey two novel bits to successfully combine the two approaches:\nFirstly a loss function in order to induce a 3D mask field on a voxel grid based on the SAM outputs (Eq. 4) and secondly (and most importantly) a method to prompt SAM on another view based on the existing 3D mask view - allowing one to only prompt from one image and then perform \"prompt propagation\" to the rest.", "questions": "-some more validation of the statement that SA3D can improve SAM would be welcome. \n\n- Self prompting strategy: I am not too sure I see the necessity of this particular prompting strategy. The authors are lifting every prompt point to 3D, reducing the mask score in 3D in its neighborhood, regenerating the mask, and picking the next strongest point. How much better is this than a plain 2D-based nonmaximum suppression strategy? This should be  more efficient computationally and yield roughly the same results - if not, how much worse are they?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Simple approach that should be easy to reproduce - the authors also share their code, which looks reasonably clean. \n\nSolid experimental improvements - I am convinced by these numbers that the method outperforms some of the latest state-of-the-art systes.\n\nGood ablations.\n\nInteresting -but a bit hand-wavy - results that SA3D can improve SAM; some more validation of this would be welcome.", "weaknesses": "- Overstatement: the authors are  using SAM in tandem with Nerf, to get 3D objects out of a scene, which is great - but practically this is similar to classic co-segmentation or the systems that they compare to, but a bit better because of piggy-backing on SAM. \nStating in the abstract that  \"Our research offers a generic and efficient methodology to lift a 2D vision foundation model to 3D, as long as the 2D model can steadily address  promptable segmentation across multiple views.\" suggests more than just this - one can start imagining extending the attention operations to 3D, supervising for depth/volumetric reconstruction etc, or more importantly having a foundation model for 3D, none of which is  the case based on what we have in the present paper. Technically the statement is not false - but implies more than what is actually happenning in the paper. \n\n\n- I could not find any discussion about the computational efficiency of the proposed algorithm. At the moment it's unclear what is the importance of this.\n\nMinor:\n\n- Unclear intuitively what the second term does in Eq. 5 from the way it's written. I think a more obvious rewrite is \n(lambda - 1) L_proj + lambda sum_r M(r) \\propto L_proj + lambda/(lambda-1) sum_r M(r), \nand explain that the second term is just a regularization term on the optimized segmentation field.\n\n- l. 280: \"we demonstrate limitations of SA3D in panoptic segmentation\" -> could not find any pointer to this in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your efforts on the review work and your detailed comments. We answer your questions as follows and hope the response could clear your concerns.\n### Weaknesses\n> W1: Overstatement: ... \"Our research ... as long as ...\" suggests more ...\n\n**A1:** Thanks for the suggestion. We acknowledge that SA3D may not have all the abilities you mentioned, but it indeed has some valuable potential. To better support our statement, we demonstrate the generalization ability of our framework with some other models (please check our **A1** to b3JM).\n\nFollowing your suggestion, we change our statement to “Our research reveals a **potential** methodology to lift **the ability** of a 2D vision foundation model to 3D, as long as the 2D model can steadily address promptable segmentation across multiple views.” We welcome any other suggestions on improving this part.\n\n> W2: ... discussion about the computational efficiency ...\n\n**A2:** Please check the global response for a comprehensive discussion about computation efficiency.\n\n> W3: Unclear ... Eq. 5 ... a more obvious rewrite ...\n\n**A3:** Thanks for the suggestion. The second term is essentially a regularization term that adds a negative effect on all regions involved in rendering the current mask to suppress the potential inaccurately-segmented region. \nOnly if SAM consistently predicts a region as foreground from different views, SA3D marks its corresponding 3D region as foreground. We will clarify the statement around Line 135-138 and take this suggested equation as a supplement for better understanding.\n\n> W4: l. 280: \"we demonstrate limitations ... \" -> could not find ...\n\n**A4:** Sorry for the misleading. Here what we want to express is we admit that SA3D has some limitations in panoptic segmentation.\nWe will revise the statement in Line 280 as “SA3D has limitations in panoptic segmentation”. Hope this can resolve the ambiguity.\n\n### Questions\n\n> Q1: ... validation of ... SA3D can improve SAM ...\n\n**A5:** To further demonstrate our statement, we provide more visualization results (Fig. 1 of the global response PDF) on the 'bonsai' scene of the Mip-NeRF 360 dataset. Here we give a detailed explanation about why “SA3D can improve SAM” for clarification.\n\nSegmentation models often face challenges in accurately capturing object details such as small holes and gaps due to limitations in resolution. Even though SAM exhibits fine-grained segmentation capabilities, this issue still persists.\n\nThe ability to improve SAM's performance stems from the fine-grained depth estimation (or the geometry information) provided by NeRF. The utilization of such information to aid segmentation has been a long-standing problem [1-3].\n\nSpecific to SA3D, the geometry information is utilized through the incorporation of a negative refinement (regularization) term in the projection loss (Eq. 5). When SAM overlooks small holes and gaps, the mask passes through these regions and gets projected onto the background behind the object. However, with a viewpoint switched, these inaccurately-segmented regions shift from being behind the target object to the side. In these new views, SAM's foreground prediction no longer includes these regions. Consequently, the mask confidence score for these regions is effectively suppressed by the negative refinement (regularization) term. We hope this explanation can help clear up the confusion.\n\nPresently, we cannot find a dataset that supports evaluation for both fine-grained segmentation and NeRF reconstruction. Consequently, we are unable to provide quantitative metrics to validate the statement. We believe that cultivating such a dataset or benchmark represents a promising avenue for future research.\n\n> Q2: Self prompting strategy: ... the necessity of this particular prompting strategy... How much better than a plain 2D-based NMS? ... yield roughly the same results - if not, how much worse ...?\n\n**A6:** Our self-prompting strategy can be treated as a variant of the NMS algorithm. Without the 3D distance based confidence score decay, it degenerates into a simple 2D NMS, which selects a prompt point with the highest confidence score and then blocks out a surrounding region of it (as shown in Line 159-160). To answer the question, we conducted an ablation experiment on the NVOS dataset:\n\n||w/ Confidence Decay Term||w/o Confidence Decay Term||\n|-|-|-|-|-|\n|Scene|IoU|Acc|IoU|Acc|\n|fern|82.9|94.4|82.9|94.4|\n|flower|94.6|98.7|94.6|98.7|\n|fortress|98.3|99.7|98.4|99.7|\n|horns_center|96.2|99.3|96.2|99.3|\n|horns_left|90.2|99.4|88.8|99.3|\n|leaves|93.2|99.6|93.2|99.6|\n|orchids|85.5|97.3|85.4|97.3|\n|trex|82.0|97.4|64.0|93.3|\n|mean|90.3|98.2|87.9|97.7|\n\nThe above table shows that for most cases, a simple NMS self-prompting is enough. But for hard cases like 'LLFF-trex' (a trex skeleton, as shown in Fig. 5 of our paper), where a large number of depth jumps, the confidence decay contributes a lot. In such a situation, inaccurate masks bleed through gaps in the foreground onto the background. If the self-prompting mechanism generates prompts on these inaccurately-segmented regions, SAM may produce plausible segmentation results that can cheat the IoU-rejection mechanism and finally the segmentation results will involve unwanted background regions.\n\nIn addition, performing the decay step only involves some basic matrix calculation, which does not result in big time consumption. Taking the 'LLFF-fern' scene as an example, replacing the self-prompting strategy with a simple NMS only decrease the 27.04 seconds cost to 25.73 seconds.\n\n[1] Couprie, Camille, et al. \"Indoor semantic segmentation using depth information.\" arXiv preprint arXiv:1301.3572 (2013).\n\n[2] Zhang, Zhenyu, et al. \"Joint task-recursive learning for semantic segmentation and depth estimation.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n\n[3] Kerr, Justin, et al. \"Lerf: Language embedded radiance fields.\" arXiv preprint arXiv:2303.09553 (2023)."}, {"review_id": "jGLlH6NDX8", "reviewer": "Reviewer_JhYG", "summary": "This paper proposes a method for segmenting a pre-trained NeRF by utilizing SAM. Given a pre-trained NeRF, it first asks users to provide prompts (e.g., some points) for a reference view. It then utilizes the SAM to generate a 2D segmentation for the reference view and utilizes the 2D mask to optimize a 3D segmentation mask. After that, it will iteratively update the 3D mask by (a) selecting a random review, (b) rendering the view, (c) utilizing the 3D mask to automatically generate prompts for the view, (d) utilizing prompts to query SAM, (e) utilizing SAM output to update 3D mask. The method proposes a self-promoting strategy to generate prompts given the 3D mask and an IoU-aware view rejection to ignore the bad prediction from SAM.", "questions": "1. How many gradient descent iterations do you need in each inverse rendering step?\n\n2. In Table 2, SA3D performs worse than the baseline method (MVSeg). Is there any common pattern for these instances? Or any reasons for the performance differences?\n", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel method for segmenting a pre-trained NeRF by using SAM.\n2. The paper proposes a self-promoting strategy for automatically generating prompts for SAM and a rejection strategy to ignore bad SAM predictions.\n3. The authors conduct experiments on three datasets and provide some ablation studies.\n4. The paper is easy to follow.", "weaknesses": "1. The method requires minutes for a single segmentation (e.g., of an object), which may greatly limit the usage of the method in many real-world applications (e.g., robotics manipulation).\n\n2. Recently, there are also many NeRF/Point Cloud-based open-world (vocabulary) 3D segmentation (for both scene-level and part-level) methods[1-8] that leverage pre-trained 2D VLM (e.g., CLIP). However, the discussion and comparison with them is missing. It seems that many of these prior methods don't need per-instance optimization and can generate a 3D segmentation mask in just seconds. Please cite these papers and discuss the advantages of the proposed methods.\n\n3. In Line 153, the paper states, \"Given an incomplete 2D rendered mask\". Why is the rendered mask always incomplete? Is it possible that some SAM predictions are wrong and include extra regions, which leads to an enlarged 3D segmentation mask? If this is possible, the self-prompting strategy will also generate a wrong prompt for SAM in the later steps. Please explain whether this case is possible and how the proposed method can handle the wrong SAM prediction (including extra regions).\n\n4. The negative refinement term (Line 137) is unclear to me. Please explain in more detail about the motivation for this term.\n\n5. Equation (7) is not clear to me. Could you explain in more detail? Also, it would be better to provide an ablation study to verify the necessity of the confidence decay step (Equation (7)). Can the self-prompting strategy still work without confidence decay?\n\n6. For all tables, could you include runtime for both the proposed method and baseline methods?\n\n7. It would be better to include evaluations on some standard 3D segmentation benchmarks (e.g., ScanNet and PartNet) to have extensive comparison with existing methods as well. \n\n\n\n[1] Kerr, Justin, et al. \"Lerf: Language embedded radiance fields.\" arXiv preprint arXiv:2303.09553 (2023).\n\n[2] Peng, Songyou, et al. \"Openscene: 3d scene understanding with open vocabularies.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Ding, Runyu, et al. \"PLA: Language-Driven Open-Vocabulary 3D Scene Understanding.\" arXiv preprint arXiv:2211.16312 (2022).\n\n[4] Ha, Huy, and Shuran Song. \"Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models.\" 6th \nAnnual Conference on Robot Learning. 2022.\n\n[5] Zhang, Junbo, Runpei Dong, and Kaisheng Ma. \"Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip.\" arXiv preprint arXiv:2303.04748 (2023).\n\n[6] Liu, Minghua, et al. \"Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[7] Yang, Jihan, et al. \"Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding.\" arXiv preprint arXiv:2304.00962 (2023).\n\n[8] Jatavallabhula, Krishna Murthy, et al. \"Conceptfusion: Open-set multimodal 3d mapping.\" arXiv preprint arXiv:2302.07241 (2023).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your instructive comments.\n\n### Weaknesses\n> W1: ... minutes for a single segmentation ...\n\n**A1:** The time cost reported in our paper is an upper bound. As shown in Table 4 (global response text), SA3D achieves satisfactory segmentation with a few sampled views, which only requires < 10 seconds. \n\n> W2: ... [1-8] that leverage VLM ...  cite and discuss ...\n\n**A2:** Thanks. We will cite all these papers and add discussions below.\n\nThe most relevant work to SA3D is LERF [1], which trains a feature field of the VLM together with the radiance field. Compared with SA3D, LERF focuses on coarsely localizing the specific objects with text prompts but not fine-grained 3D segmentation. The reliance on CLIP features makes it insensitive to the specific location information of the target object. When there are multiple objects with similar semantics in the scene, LERF cannot perform effective 3D segmentation. We evaluate the segmentation ability of LERF on the NVOS dataset and attach the results in the global response PDF. Moreover, we also provide the visualization results in Fig 6 of the global response PDF to support our statement.\n\nThe remaining methods mainly focus on point clouds. By connecting the 3D point cloud with specific camera poses with 2D multi-view images, the extracted features by VLM can be projected to the 3D point cloud. The data acquisition of these methods is more expensive than ours, i.e., acquiring multi-view images for NeRFs.\n\n> W3: Line 153 ... rendered mask always incomplete? ... handle the wrong SAM prediction？\n\n**A3:** Sorry for the ambiguous statement \"incomplete 2D rendered mask\". We will replace \"incomplete\" with \"inaccurate\".\n\nIt is possible that SAM generates inaccurate predictions. We designed several mechanisms to tackle this problem:\n- The negative refinement term in the mask inverse rendering loss (Eq. 5). In each iteration, if a region is predicted as background by SAM, its mask confidence score is then suppressed, which significantly alleviates inaccurate segmentation by SAM under certain views.\n- The confidence decay term in the self-prompting strategy (Eq. 8). This term adjusts rendered mask confidence scores according to the distance in the 3D space between the selected prompts and the candidate coordinates. This facilitates closely prompted points in 3D.\n- The IoU-aware view rejection (Lines 171-176). When SAM predicts a mask that greatly differs from the currently rendered one, i.e., obtaining a low IoU, this view will be skipped to avoid wrong mask allocation.\n\n> W4: The negative refinement term ... unclear ...\n\n**A4:** This term is used to suppress the extra regions included in SAM predictions on the mask grids: When a region is not segmented as foreground by SAM, its mask confidence score is suppressed. Thus the mask grids are labeled as foreground only if the region is consistently classified as foreground in different views. We will clarify it in Lines 135-138.\n\n> W5: Eq. (7) not clear ... ablation study...?\n\n**A5:** Eq. 7 defines a confidence decay term for self-prompting, based on the intuition that prompt points should not be too far apart in the 3D space. Assuming we have gathered N prompt points. When we try to select the (N+1)-th point from the remaining candidates, we first check the distance between these candidates and the existing N prompt points. If a candidate is far from all of them, the confidence score is suppressed heavily. \n\nTo realize this, for a candidate, we traverse the existing prompts and get a set of decay terms. The smallest of them is determined as the final decay term for the candidate. This decay term between a candidate and an existing prompt point is defined as a product involving the confidence score of the prompt point and the min-max normalized 3D Euclidean distance between the two points.\n\nIn common cases, without the decay step, the self-prompting still works. But for some hard cases, it may fail. For the concrete ablation and discussion, please refer to our response to R-Xgwo's Q2.\n\n> W6: ... runtime ...\n\n**A6:** See the global response texts for the time cost.\n\n> W7: ... evaluations on some standard 3D segmentation benchmarks ...\n\n**A7:** Please kindly note SA3D performs interactive segmentation, which is quite different from traditional 3D segmentation approaches. However, making SA3D support traditional 3D segmentation is an interesting research direction. We still provide the experimental results on Scannet for a more comprehensive evaluation. The experimental setting follows Table 3. Some results (mIoU) are shown as follows:\n\n||scannet0050_02|scannet0144_01|scannet0300_01|scannet0354_00|scannet0389_00|\n|-|-|-|-|-|-|\n|Single View|53.0|63.5|61.2|56.9|60.6|\n|SA3D|72.9|77.1|75.8|69.8|78.9|\n\n### Questions\n\n> Q1: ... iterations ... in each inverse rendering step?\n\n**A8:** Only one gradient descent iteration is required.\n\n> Q2: In Table 2, SA3D performs worse than ... MVSeg.\n\n**A9:** Yes. This gap stems from the inductive bias of SAM and some ambiguous segmentation targets. We provide some visualization results to support the statement in Fig 5 (global response PDF).\nIn the 'Orchids' scene, the segmentation target is a group of flowers. SAM tends to segment each flower separately. If SAM is forced to segment them into one, it may involve some unexpected regions in the prediction.\nA similar phenomenon happens in the 'Room' scene. The SPIn-NeRF dataset treats the table along with objects on it as a whole. SAM segments the table separately, ignoring the placed objects.\nFor the 'Pinecone' scene, both MVSeg and SA3D perform well. This scene involves many details, which are ignored by the ground-truth sometimes. We believe a 0.5% mIoU gap is reasonable since both our segmentation results and corresponding annotations are not perfect."}, {"review_id": "7TF8cx1tOz", "reviewer": "Reviewer_b3JM", "summary": "This paper proposes to lift 2D segmentations from foundation models such as SAM to 3D by iterating between SAM and NeRF, without re-training or re-defining either. Given a trained NeRF model, the model first renders a view, which is also processed by SAM given a user click. With the segmentation by SAM, the model optimizes a 3D segmentation volume such that it volume renders into a mask consistent with what SAM produced (“mask inverse rendering”). Next, the model projects this initial segmentation volume to other viewpoints, producing incomplete 2D masks. Finally, the model computes “good prompts” for SAM to complete these masks (“cross-view self prompting”). The model iterates between these steps until a complete segmentation volume has been produced.\n\nBecause of the generalization power of SAM, the model is able to segment almost anything in 3D, without requiring applying changes to SAM or NeRF, making itself a framework that can be applied to any 2D foundation models that we want to lift to 3D.\n", "questions": "\nHas runtime been reported? I recommend augmenting Table 4 with an additional row of runtime.\n\n\n", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "\nThe proposed method is simple yet effective, following the recent trend of developing foundation models and/or open-vocabulary LLMs. More importantly, it also bridges the gap between powerful 2D models and 3D understanding as required by robots or autonomous vehicles. It is general and applicable to any 2D foundation models. The results are strong both qualitatively and quantitatively.\n\n", "weaknesses": "Since the framework is claimed to be (and I think it is) general and applicable to any 2D foundation models, the paper will be much stronger if the authors could demonstrate the use of this framework to lift another foundation model’s output into 3D. \n\nThe paper will also benefit from showing some 3D shape results that are extracted from the 3D segmentation volume. Often, it’s the underlying 3D geometry that matters for, say, robotic manipulation. “RGB looking good” is separated than that.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable feedback and hope our following clarifications and responses could clear your concerns.\n### Weaknesses\n\n> W1: Demonstrate the use of this framework to lift another foundation model’s output into 3D.\n\n**A1:** Thanks for the suggestion. We try to lift SEEM [1] with SA3D, which is a concurrency work of SAM. As suggested by Reviewer jjUN, we also lift three interactive methods with SA3D. SEEM supports more modals but suffers from relatively worse segmentation performance. We evaluate the performance of SA3D with SEEM (and the three other interactive methods) on the NVOS dataset. The results are shown as below:\n\n|              | SEEM       |        | SimpleClick |           | RITM      |           | FocalClick |           |\n|:------------:|------------|--------|-------------|-----------|-----------|-----------|------------|-----------|\n|     Scene    | IoU        | Acc    | IoU         | Acc       | IoU       | Acc       | IoU        | Acc       |\n|     fern     | 0.7921     | 0.9293 | 0.8109      | 0.9347    | 0.7662    | 0.9154    | 0.8214     | 0.9404    |\n|    flower    | 0.9313     | 0.9836 | 0.9362      | 0.9849    | 0.8912    | 0.9728    | 0.9372     | 0.9852    |\n|   fortress   | 0.9851     | 0.9972 | 0.9761      | 0.9954    | 0.9755    | 0.9953    | 0.9839     | 0.997     |\n| horns_center | 0.8182     | 0.9627 | 0.9554      | 0.9921    | 0.8642    | 0.9759    | 0.9547     | 0.9920     |\n|  horns_left  | -          | -      | 0.8599      | 0.9908    | 0.8451    | 0.9896    | 0.8407     | 0.9894    |\n|    leaves    | -          | -      | 0.9326      | 0.9957    | 0.9341    | 0.9958    | 0.9258     | 0.9952    |\n|    orchids   | 0.8655     | 0.9771 | 0.7869      | 0.9636    | 0.5744    | 0.9089    | 0.8984     | 0.9827    |\n|     trex     | 0.7700       | 0.9701 | 0.7598      | 0.9647    | 0.6418    | 0.9472    | 0.7507     | 0.9636    |\n|     mean     | 0.8604 | 0.9700   | 0.8772    | 0.9778 | 0.8116 | 0.9626 | 0.8891     | 0.9807 |\n\nPlease kindly note that the missing entries for SEEM are because it cannot generate reasonable segmentation results for the reference view images no matter how we adjust the prompts. Some visualization results of SEEM are provided in Fig. 2 of the global author response PDF. SEEM has the advantage of accommodating multiple cross-modal prompt inputs, but exhibits inferior segmentation performance compared to SAM. Further utilizing the cross-modal prompting ability of different foundation models to enhance the behaviour of self-prompting is a promising direction.\n\n> W2: Showing some 3D shape results that are extracted from the 3D segmentation volume.\n\n**A2:** We extract meshes from the segmented 3D objects based on the standard marching cubes algorithm. Visualization results can be found in Figure 4 of the global author response PDF. Please note thatthe quality of these meshes can be further improved by applying more effective NeRF2Mesh methods [2][3]. The scripts for mesh extraction will also be released to facilitate the following research.\n\n### Questions\n\n> Q1: Augmenting Table 4 with an additional row of runtime.\n\n**A3:** We update Table 4 with runtime supplemented as follows. It shows a trade-off between time cost (related to the number of views) and the segmentation quality.\n\n| |||||\n|----------------------------------|----------|----------|----------|------------|\n| Number of Views                  | 5 (10%)  | 9 (20%)  | 21 (50%) | 43 (100%)  |\n| IoU on Fortress (forward facing) | 97.8     | 98.3     | 98.3     | 98.3       |\n| Time Cost(s)                       | 7.56    | 12.80   | 28.98   | 58.97     |\n| Number of Views                  | 11 (10%) | 21 (20%) | 51 (50%) | 103 (100%) |\n| IoU on Lego (360 degrees)        | 84.5     | 84.8     | 91.5     | 92.2       |\n| Time Cost(s)                        | 23.49    | 43.54    | 103.83   | 204.93     |\n\n\n[1] Zou, Xueyan, et al. \"Segment everything everywhere all at once.\" arXiv preprint arXiv:2304.06718 (2023).\n\n[2] Tang, Jiaxiang, et al. \"Delicate textured mesh recovery from nerf via adaptive surface refinement.\" arXiv preprint arXiv:2303.02091 (2023).\n\n[3] Yariv, Lior, et al. \"BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis.\" arXiv preprint arXiv:2302.14859 (2023)."}, {"review_id": "r2w30Zn9SR", "reviewer": "Reviewer_jjUN", "summary": "This paper introduces a simple and efficient method for general 3D segmentation based on SAM (a powerful 2D foundation model) and NeRF-style representation. Instead of building a 3D foundation model from scratch, SA3D uses two steps to lifts 2D SAM segmentation results to 3D in a more concise and efficient way. Based on a well-trained NeRF, a rendered reference view and the human input prompts are sent into SAM to get the first segmentation. Then the proposed mask inverse rendering and cross-view self-prompting strategy will help optimize a volume-based 3D mask field and propagate segmentation information to different views in an iterative and incremental manner. The comprehensive experiments prove the effectiveness of the designed pipeline.", "questions": "1.\tIt seems that the method is not restricted by NeRF-based representation. Since the authors assume NeRF provides good depth, the proposed method should also work under other representations such as mesh and point cloud?\n2.\tThe 3D mask is modeled as dense voxel grid, which may cause high memory consumption for high resolution representation. How will different resolutions affect the performance? Maybe using a MLP is a better choice?\n", "limitations": "Yes. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe proposed method is simple but effective. It provides a general interactive 3D object segmentation paradigm which do not rely on heavy pre-training.\n2.\tThe method is pretty efficient. Given a pre-trained NeRF, the 3D segmentation can be completed within only minutes with the help of SAM.\n3.\tThe experiments have covered variant datasets and comparison to SOTA methods, which are comprehensive and persuasive.\n4.\tThe paper is well-written and easy to follow.\n", "weaknesses": "1.\tI expect that segment “anything” in 3D means the method can segment all the things in a 3D scene, so that it is consistent with the purpose of SAM. But the proposed SA3D can only segment one object at each time.\n2.\tFurthermore, using prompts (such as points, scribbles, text) to achieve 2D segmentation of one target object is a long-studied subject. Besides SAM, there should be many alternative choices such as [1][2][3], which should have been evaluated for this 3D task. Or the irreplaceability of SAM needs to be explained.\n3.\tI wondering if the method requires complete observation of the target object in at least one image. What if the target object is not observed completely in any views? How will the choice of reference view affect the 3D segment results? Is there any necessary strategy to achieve best performance? The robustness of the method should be evaluated.\n\n[1] Sofiiuk K, Petrov I A, Konushin A. Reviving iterative training with mask guidance for interactive segmentation[C]//2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2022: 3141-3145.\n[2] Liu Q, Xu Z, Bertasius G, et al. SimpleClick: Interactive image segmentation with simple vision transformers[J]. arXiv preprint arXiv:2210.11006, 2022.\n[3] Chen X, Zhao Z, Zhang Y, et al. Focalclick: Towards practical interactive image segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 1300-1309.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments. We answer your questions as follows and hope the response could clear your concerns.\n\n### Weaknesses\n\n> W1: ... SA3D can only segment one object at a time.\n\n**A1:** SAM achieves “segment everything” by densely prompting on the image. These prompts are stacked at the batch dimension and fed to the model. Such implementation, in essence, is equivalent to querying the SAM decoder for many objects one by one.\n\nSA3D also inherits the ability and mechanism. Segmenting multiple objects can be achieved by stacking multiple 3D mask grids into a 4D data structure (i.e. HxWxL -> NxHxWxL), where N denotes the number of objects. Then, iterative self-prompting and inverse rendering can be performed for all objects simultaneously. We attach some visualization results in Fig 3 (global response PDF).\n\nBy the way, as stated in Lines 282-285, we admit that the current method still has limitations in segmenting everything. For example, it is difficult for SAM to guarantee cross-view consistency for extremely small parts, because some parts may be segmented into different instances with similar semantics in different views. Fixing these cases in 3D is a challenging issue and is left for future research.\n\n> W2: ... Besides SAM, alternative choices [1-3] ...\n\n**A2:** Yes. SA3D can generalize to these methods effortlessly, which demonstrates our claim in the abstract: \"... lift a 2D vision foundation model to 3D, as long as the 2D model can steadily address promptable segmentation across multiple views\". We conduct experiments on the NVOS dataset and the 'Replica office 0' scene to evaluate SA3D with these interactive segmentation methods:\n\n||SAM||SimpleClick||RITM||FocalClick||\n|:-:|-|-|-|-|-|-|-|-|\n|Metrics|IoU|Acc|IoU|Acc|IoU|Acc|IoU|Acc|\n|Mean|90.3|98.2|87.7|0.9778|81.2|96.3|88.9|98.1|\n\n|Replica_office_0|SAM|SimpleClick|RITM|FocalClick|\n|-|-|-|-|-|\n|mIoU|84.4|72.7|69.0|63.6|\n\nFrom the above results we find lifting these interactive segmentation methods with SA3D can beat the previous SOTA (ISRF, mIoU 83.8; NVOS, mIoU 70.1) on the NVOS benchmark. Additionally, when encountered with complicated indoor scenes like Replica, other methods perform much worse, showing the robustness of SA3D as a foundation model.\n\n> W3: What if not observed completely in any views? ... the choice of reference view ...? ... necessary strategy ...? robustness ...\n\n**A3:** There are two cases: part of the object does not appear in some views (but appear in other views), or it does not appear at all. For the former case, SA3D can recover it using information from other views; for the latter case, there can be parts missing in the target object (see Fig 9, global response PDF). This is an interesting future direction (e.g. applying generative models such as diffusion models).\n\nTo evaluate the robustness of SA3D, we randomly select 3 new reference views from the training set and compute the mean metrics on the NVOS dataset. The results are as follows:\n\n||Random 3 views||Reported in the Paper||\n|-|-|-|-|-|\n|Metrics|IoU|Acc|IoU|Acc|\n|Mean|89.8|98.2|90.3|98.2|\n\nVisualization results are provided to demonstrate the robustness of SA3D to reference view. Please check Fig 8 (global response PDF).\n\n### Questions\n\n> Q1: ... not restricted by NeRF ... mesh and point cloud?\n\n**A4:** Yes. Extending SA3D to other 3D formats is straightforward. Here we elaborate a pipeline for point clouds:\n1. Generate multi-view images on the point cloud data using off-the-shelf methods (e.g. Point2Pix, CVPR'23). Practically, we find projecting 3D points with RGB information onto a 2D plane is enough for SAM to segment\n2. Select a reference view and input prompts; use SAM for segmentation\n3. Assign the predicted mask to the corresponding 3D points (Mask Inverse Rendering)\n4. Project the 3D mask onto a new view and get the corresponding 2D image (Mask Rendering)\n5. Extract self-prompting points from the 2D mask (Self-prompting)\n6. Feed the prompt and 2D image to SAM for segmentation\n7. Repeat 3-6 until getting the full object\n\n> Q2: ... dense voxel grid ... high memory consumption ... different resolutions affect ... MLP is better?\n\n**A5:** Good question! In our implementation, the resolution of the mask grids is set to be TensoRF grids (320^3). We study different resolutions of mask grids on the NVOS dataset, results shown below and visualization provided in Fig 7 (global response PDF). The effect of the grid resolution is slight.\n\n|$320^3$||$160^3$||$80^3$||$40^3$||MLP||\n|-|-|-|-|-|-|-|-|-|-|\n|mIoU|mAcc|mIoU|mAcc|mIoU|mAcc|mIoU|mAcc|mIoU|mAcc|\n|90.3|98.2|90.1|98.2|89.2|98.1|88.0|97.8|80.9|96.4|\n\nWe also report the results of an MLP version (not better) in the above table. Below are the advantages of explicit mask grids over MLP.\n\n1. Optimization for mask grids is explicit and straightforward. In contrast, assigning a 3D point as positive in an MLP may unexpectedly make other points positive in the 3D space, leading to unstable self-prompting.\n2. In gradient descent, using explicit mask grids saves a lot of memory because its computation graph is simple. For MLP, the computation graph contains too many non-leaf nodes for different layers of the MLP.\n3. By adopting mask grids, the 3D segmentation results are explicit, which makes the downstream task, *e.g.* editing, more convenient. For example, removing the target object or extracting it from the scene only requires direct matrix multiplication between the mask grids and the density grids (if the density grids are also stored explicitly).\n\nWe agree that MLP enjoys theoretically infinite resolution and smaller storage costs. Therefore, exploring hybrid mask representations is a promising future direction."}]

Author rebuttal:
## Common Response for Time Cost Analysis

We thank all reviewers for the insightful comments. 

Since several reviewers (Q1 of Reviewer b3JM, W1 & W6 of Reviewer JhYG, W2 of Reviewer Xgwo) express concerns about the time overhead of our method, we discuss this issue here.

We provide per-scene time cost for the LLFF dataset and some unbounded scenes involved in Table 2. Please kindly note that the experiments of Table 1 and Table 2 are both based on the scenes of the LLFF dataset. Consequently, we omit the time cost for Table 1. Besides, the average time cost (seconds) comparisons for each object of the Replica dataset are shown in Table 3. Table 4 supplements time cost analysis regarding the number of views to illustrate the trade-off between time cost and accuracy.

The modified tables and attached table are shown as follows.

---
**Table 1. Comparison with ISRF**

There are three methods involved in the comparison in Table 1: Graph Cut (3D), NVOS and ISRF. The first two methods neither provide the code to reproduce nor clarifies the time overhead in their paper. ISRF provide rough time cost estimations. Thus we compare with ISRF roughly for segmenting an object in a scene:

| ISRF || SA3D | |
|:-:|:-:|:-:|:-:|
| Step |Time Cost | Step |Time Cost |
|||User Intervention (One Time) |-|
|||Initial Segmentation |< 1 second |
| Training feature field | 2.5 minutes |Training Mask Grids | 10 seconds - 3 minutes|
|User Intervention (Many Times)| - || |
| K-Means Clustering | 2 seconds |||
| 3D Feature Query | 1 seconds |||
| Bilateral Region Growing | 0.3 seconds|||

The coarse time cost of ISRF is gathered from their paper. For segmenting an object, ISRF requires many iterations of user intervention (and the following steps). Compared with ISRF, SA3D takes similar or smaller time cost but enjoys a more concise procedure. Please note that for both methods the time cost of pre-training a NeRF is omitted for clear comparisons.

---
**Table 2. Comparisons on LLFF scenes and some 360 degrees scenes**

|Scenes|Single View|||MVSeg|||SA3D (ours)|||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
||IoU(%)|Acc(%)|Time Cost(s)|IoU(%)|Acc(%)|Time Cost(s)|IoU(%)|Acc(%)|Time Cost(s)|
|Orchids|79.4|96.0|-|92.7|98.8|264|83.6|96.9|33.9|
|Leaves|78.7|98.6|-|94.9|99.7|271|97.2|99.9|30.1|
|Fern|95.2|99.3|-|94.3|99.2|246|97.1|99.6|26.1|
|Room|73.4|96.5|-|95.6|99.4|284|88.2|98.3|52.7|
|Horns|85.3|97.1|-|92.8|98.7|276|94.5|99.0|75.3|
|Fortress|94.1|99.1|-|97.7|99.7|273|98.3|99.8|46.8|
|Fork|69.4|98.5|-|87.9|99.5|244|89.4|99.6|27.7|
|Pinecone|57.0|92.5|-|93.4|99.2|257|92.9|99.1|73.4|
|Truck|37.9|77.9|-|85.2|95.1|261|90.8|96.7|333.5|
|Lego|76.0|99.1|-|74.9|99.2|263|92.2|99.8|180.9|
|mean|74.6|95.5|-|90.9|98.9|264|92.4|98.9|88.0|

Note that as we introduced in Line 215-217, "Single View" denotes directly projecting the mask of the reference view onto the mask grids without any following operations, which takes almost no time cost. Thus we omit its time cost in Table 2 and Table 3.

---
**Table 3. Comparisons on Replica**

|Methods|metrics|office0|office1|office2|office3|office4|room0|room1|room2|mean|
|:-:|:-:|-|-|-|-|-|-|-|-|-|
|Single View|mIoU(%)|68.7|56.5|68.4|62.2|57.0|55.4|53.8|56.7|59.8|
||Time Cost(s/object)|-|-|-|-|-|-|-|-|-|
|MVSeg|mIoU(%)|31.4|40.4|30.4|30.5|25.4|31.1|40.7|29.2|32.4|
||Time Cost(s/object)|1567|1360|1343|1617|1301|1292|1431|1527|1430|
|SA3D(ours)|mIoU(%)|84.4|77.0|88.9|84.4|82.6|77.6|79.8|89.2|83.0|
||Time Cost(s/object)|81.2|53.9|46.4|62.1|76.4|58.1|61.5|101.2|67.6|

MVSeg does not report the time cost in their paper. We reproduce MVSeg with its official code on the Replica dataset to measure its time cost. As shown in the modified Table 3, MVSeg takes a high time cost as MVSeg needs to train a Semantic-NeRF for each target in its official implementation. 

---
**Table 4. Ablation on different numbers of views**

||||||
|-|-|-|-|-|
|Number of Views|5(10%)|9(20%)|21(50%)|43(100%)|
|IoU on Fortress (forward facing)|97.8|98.3|98.3|98.3|
|Time Cost(s)|7.56|12.80|28.98|58.97|
|Number of Views|11(10%)|21(20%)|51(50%)|103(100%)|
|IoU on Lego (360 degrees)|84.5|84.8|91.5|92.2|
|Time Cost(s)|23.49|43.54|103.83|204.93|

Please kindly note that the time cost of SA3D depends on the number of views during the mask training phase. Consequently, there exist differences in the time overhead of SA3D between different scenes. Nonetheless, on the whole, the time overhead of SA3D typically remains within 1 minute.

---
We also provide a possible accelerating solution for future updating. Except for the first reference view, the following optimization can be parallelized. In each iteration, SA3D can select multiple new views and conduct self-prompting and inverse rendering for them simultaneously. Gradients on mask grids can then be gathered for optimization. 

We hope the discussion can address the concern of time consumption.

Meta-review: This paper proposes to lift 2D predictions by SAM to the 3D space based on NeRF. The initial reviews are generally positive. The rebuttal further clarified concerns raised by reviewers. In agreement with the reviewers, AC recommends acceptance since the proposed method is simple yet effective.
Decision: Accept (poster)