Title: Segment Anything in 3D with NeRFs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Segmentation

NeRF

3D segmentation
Abstract: Recently, the Segment Anything Model (SAM) emerged as a powerful vision foundation model which is capable to segment anything in 2D images. This paper aims to generalize SAM to segment 3D objects. Rather than replicating the data acquisition and annotation procedure which is costly in 3D, we design an efficient solution, leveraging the Neural Radiance Field (NeRF) as a cheap and off-the-shelf prior that connects multi-view 2D images to the 3D space. We refer to the proposed solution as SA3D, for Segment Anything in 3D. It is only required to provide a manual segmentation prompt (e.g., rough points) for the target object in a single view, which is used to generate its 2D mask in this view with SAM. Next, SA3D alternately performs mask inverse rendering and cross-view self-prompting across various views to iteratively complete the 3D mask of the target object constructed with voxel grids. The former projects the 2D mask obtained by SAM in the current view onto 3D mask with guidance of the density distribution learned by the NeRF; The latter extracts reliable prompts automatically as the input to SAM from the NeRF-rendered 2D mask in another view. We show in experiments that SA3D adapts to various scenes and achieves 3D segmentation within minutes. Our research offers a generic and efficient methodology to lift a 2D vision foundation model to 3D, as long as the 2D model can steadily address promptable segmentation across multiple views.

Segment Anything in 3D with NeRFs

Abstract

Recently, the Segment Anything Model (SAM) emerged as a powerful vision foundation model which is capable to segment anything in 2D images. This paper aims to generalize SAM to segment 3D objects. Rather than replicating the data acquisition and annotation procedure which is costly in 3D, we design an efficient solution, leveraging the Neural Radiance Field (NeRF) as a cheap and off-theshelf prior that connects multi-view 2D images to the 3D space. We refer to the proposed solution as SA3D , for Segment Anything in 3D. It is only required to provide a manual segmentation prompt ( e.g. , rough points) for the target object in a single view , which is used to generate its 2D mask in this view with SAM. Next, SA3D alternately performs mask inverse rendering and cross-view selfprompting across various views to iteratively complete the 3D mask of the target object constructed with voxel grids. The former projects the 2D mask obtained by SAM in the current view onto 3D mask with guidance of the density distribution learned by the NeRF; The latter extracts reliable prompts automatically as the input to SAM from the NeRF-rendered 2D mask in another view. We show in experiments that SA3D adapts to various scenes and achieves 3D segmentation within minutes. Our research reveals a potential methodology to lift the ability of a 2D vision foundation model to 3D, as long as the 2D model can steadily address promptable segmentation across multiple views. Our code is available at https://github.com/Jumpat/SegmentAnythingin3D .

1 Introduction

The computer vision community has been pursuing a vision foundation model that can perform basic tasks ( e.g. , segmentation) in any scenario and for either 2D or 3D image data. Recently, the Segment Anything Model (SAM) [25] emerged and attracted a lot of attention, due to its ability to segment anything in 2D images, but generalizing the ability of SAM to 3D scenes remains mostly uncovered. One may choose to replicate the pipeline of SAM to collect and semi-automatically annotate a large set of 3D scenes, but the costly burden seems unaffordable for most research groups.

We realize that an alternative and efficient solution lies in equipping the 2D foundation model ( i.e. , SAM) with 3D perception via a 3D representation model. In other words, there is no need to establish a 3D foundation model from scratch. However, there is a prerequisite: the 3D representation model shall be capable to render 2D views and register 2D segmentation results to the 3D scene. Thus, we use the Neural Radiance Fields (NeRF) [38, 53, 3] as an off-the-shelf solution. NeRF is a family of algorithms that formulates each 3D scene into a deep neural network that serves as a 3D prior connecting multiple 2D views.

1

st

step

Render a view

NeRF

As shown in Figure 1, our solution is named Segment Anything in 3D ( SA3D ). Given a NeRF trained on a set of 2D images, SA3D takes prompts ( e.g. , click points on the object) in a single rendered view as input, which is used to generate a 2D mask in this view with SAM. Next, SA3D alternately performs two steps across various views to iteratively complete the 3D mask of the object constructed with voxel grids. In each round, the first step is mask inverse rendering , in which the previous 2D segmentation mask obtained by SAM is projected onto the 3D mask via density-guided inverse rendering offered by the NeRF. The second step is cross-view self-prompting , in which NeRF is used to render the 2D segmentation mask (which may be inaccurate) based on the 3D mask and the image from another view, then a few point prompts are automatically generated from the rendered mask and fed into SAM to produce a more complete and accurate 2D mask. The above procedure is executed iteratively until all necessary views have been sampled.

We conduct various ( e.g. , object, part-level) segmentation tasks on the Replica [51] and NVOS [47] datasets. Without re-training/re-designing SAM or NeRF, SA3D easily and efficiently adapts to different scenarios. Compared to existing approaches, SA3D enjoys a simplified pipeline that typically completes 3D segmentation within minutes. SA3D not only offers an efficient tool for segmenting anything in 3D, but also reveals a generic methodology to lift 2D foundation models to the 3D space. The only prerequisite lies in the ability to steadily address promptable segmentation across multiple views, and we hope it becomes a general property of 2D foundation models in the future.

2 Related Work

2D Segmentation Since FCN [36] has been proposed, research on 2D image segmentation has experienced a rapid growth. Various sub-fields of segmentation have been explored deeply by numerous studies [18, 24, 4, 71]. With transformers [58, 10] entering the field of segmentation, many new segmentation architectures [72, 7, 6, 52, 63] have been proposed and the whole field of segmentation has been further developed. A recent significant breakthrough in this field is the Segment Anything Model (SAM) [25]. As an emerging vision foundation model, SAM is recognized as a potential game-changer, which aims to unify the 2D segmentation task by introducing a promptbased segmentation paradigm. An analogous model to SAM is SEEM [75], which also exhibits impressive open-vocabulary segmentation capabilities.

3D Segmentation Numerous methods have explored various types of 3D representations to perform 3Dsegmentation. These scene representations include RGB-D images [60, 62, 64, 8], point clouds [44, 45, 70] and grid space such as voxels [21, 55, 35], cylinders [74] and bird's eye view space [67, 16]. Although 3D segmentation has been developed for a period of time, compared with 2D segmentation, the scarcity of labeled data and high computational complexity make it difficult to design a unified framework similar to SAM.

Lifting 2D Vision Foundation Models to 3D To tackle the limitation of data scarcity, many previous studies [23, 43, 9, 17, 69, 31, 65, 22] explored lifting 2D foundation models to 3D. In these studies, the most relevant work to SA3D is LERF [23], which trains a feature field of the Vision-Language Model ( i.e. , CLIP [46]) together with the radiance field. Compared with SA3D,

LERF focuses on coarsely localizing the specific objects with text prompts but not fine-grained 3D segmentation. The reliance on CLIP features makes it insensitive to the specific location information of the target object. When there are multiple objects with similar semantics in the scene, LERF cannot perform effective 3D segmentation. The remaining methods mainly focus on point clouds. By connecting the 3D point cloud with specific camera poses with 2D multi-view images, the extracted features by 2D foundation models can be projected to the 3D point cloud. The data acquisition of these methods is more expensive than ours, i.e. , acquiring multi-view images for NeRFs.

Segmentation in NeRFs Neural Radiance Fields (NeRFs) [38, 53, 3, 1, 40, 19, 13, 61, 30, 12] are a series of 3D implicit representation. Inspired by their success in 3D consistent novel view synthesis, numerous studies have delved into the realm of 3D segmentation within NeRFs. Zhi et al. [73] proposes Semantic-NeRF, a method that incorporates semantics into appearance and geometry. They showcase the potential of NeRFs in label propagation and refinement. NVOS [47] introduces an interactive approach to select 3D objects from NeRFs by training a lightweight multilayer perception (MLP) using custom-designed 3D features. Other approaches, e.g. N3F [57], DFF [27], LERF [23] and ISRF [15], aim to lift 2D visual features to 3D through training additional feature fields. These methods are required to re-design/-train NeRF models and usually involve additional feature-matching processes. There are also some other instance segmentation and semantic segmentation approaches [50, 41, 11, 68, 34, 20, 2, 14, 59, 28] combined with NeRFs.

The most closely related approach to our SA3D is MVSeg [39], a component of SPIn-NeRF [39], which focuses on NeRF inpainting. MVSeg adopts video segmentation techniques to propagate a 2D mask across different views and employs these masks as labels for training a Semantic-NeRF model. However, video segmentation models lack explicit 3D structure information, which are hard to handle significant occlusions in complex scenes. Our method aims at building NeRF-driven consistency across views based on self-prompting and lifting 2D masks to robust 3D masks.

3 Method

In this section, we first give a brief review of Neural Radiance Fields (NeRFs) and the Segment Anything Model (SAM). Then we introduce the overall pipeline of SA3D. Finally, we demonstrate the design of each component in SA3D in detail.

3.1 Preliminaries

Neural Radiance Fields (NeRFs) Given a training dataset I of multi-view 2D images, NeRFs [38] learn a function f θ : ( x , d ) → ( c , σ ) , which maps the spatial coordinates x ∈ R 3 and the view direction d ∈ S 2 of a point into the corresponding color c ∈ R 3 and volume density σ ∈ R . θ denotes the learnable parameters of the function f which is usually represented by a multi-layer perceptron (MLP). To render an image I θ , each pixel undergoes a ray casting process where a ray r ( t ) = x o + t d is projected through the camera pose. Here, x o is the camera origin, d is the ray direction, and t denotes the distance of a point along the ray from the origin. The RGB color I θ ( r ) at the location determined by ray r is obtained via a differentiable volume rendering algorithm:

where ω ( r ( t )) = exp( -∫ t t n σ ( r ( s ))d s ) · σ ( r ( t )) , and t n and t f denote the near and far bounds of the ray, respectively.

Segment Anything Model (SAM) SAM [25] takes an image I and a set of prompts P as input, and outputs the corresponding 2D segmentation mask M SAM in the form of a bitmap, i.e.,

The prompts p ∈ P can be points, boxes, texts, and masks.

3.2 Overall Pipeline

We assume that we already have a NeRF model trained on the dataset I . Throughout this paper, unless otherwise specified, we opt to employ the TensoRF [3] as the NeRF model, considering its

SAM

NeRF

SAM

Input Prompt

Cross-view

Self-prompting

SAM

Mask

Rendering

Image Rendering

Cross-view

Self-prompting

Mask

Rendering

view

view

Completing

view

view

3D Mask

Mask

Inverse Rendering

Mask

Inverse Rendering

Mask

Inverse Rendering

Tin

SAM

M(2)

(2)

SAM

(N)

(N)

SAM

superior efficiency of training and rendering. As shown in Figure 2, an image I in from a specific view is first rendered with the pre-trained NeRF model. A set of prompts ( e.g. , in this paper, we often use a set of points), P in , is introduced and fed into SAM along with the rendered image. The 2D segmentation mask M in SAM of the according view is obtained, which is then projected onto the 3D mask V ∈ R 3 constructed voxel grids with the proposed mask inverse rendering technique (Section 3.3). Then a 2D segmentation mask M ( n ) from a novel view is rendered from the 3D mask. The rendered mask is usually inaccurate. We propose a cross-view self-prompting method (Section 3.4) to extract point prompts P ( n ) from the rendered mask and further feed them into SAM. Thus a more accurate 2D mask M ( n ) SAM in this novel view is produced and also projected onto the voxel grids to complete the 3D mask. The above procedure is executed iteratively with more views traversed. Meanwhile, the 3D mask become more and more complete. The whole process bridges 2D segmentation results with 3D ones efficiently. Noting that no neural network needs to be optimized except the 3D mask.

3.3 Mask Inverse Rendering

As shown in Equation (1), the color of each pixel in a rendered image is determined by a sum of weighted colors along the corresponding ray. The weight ω ( r ( t )) reveals the object structure within the 3D space, where a high weight indicates the corresponding point close to the object's surface. Mask inverse rendering aims to project the 2D mask to the 3D space to form the 3D mask based on these weights.

Formally, the 3D mask is represented as voxel grids V ∈ R L × W × H , where each grid vertex stores a zero-initialized soft mask confidence score. Based on these voxels grids, each pixel of the 2D mask from one view is rendered as

where r ( t ) is the ray casting through the mask pixel, ω ( r ( t )) is inherited from density values of the pre-trained NeRF, and V ( r ( t )) denotes the mask confidence score at the location r ( t ) obtained from voxel grids V 1 . Denote M SAM ( r ) as the corresponding mask generated by SAM. When M SAM ( r ) = 1 ,

the goal of mask inverse rendering is to increase V ( r ( t )) with respect to ω ( r ( t )) . In practice, this can be optimized using the gradient descent algorithm. For this purpose, we define the mask projection loss as the negative product between M SAM ( r ) and M ( r ) :

where R ( I ) denotes the ray set of the image I .

The mask projection loss is constructed based on the assumption that both the geometry from the NeRF and the segmentation results of SAM are accurate. However, in practice, this is not always the case. We append a negative refinement term to the loss to optimize the 3D mask grids according to multi-view mask consistency:

where λ is a hyper-parameter to determine the magnitude of the negative term. With this negative refinement term, only if SAM consistently predicts a region as foreground from different views, SA3D marks its corresponding 3D region as foreground. In each iteration, the 3D mask V is updated via V ← V -η ∂ L proj ∂ V with gradient descent, where η denotes the learning rate.

3.4 Cross-view Self-prompting

Mask inverse rendering enables projecting 2D masks into the 3D space to form the 3D mask of a target object. To construct accurate 3D mask, substantial 2D masks from various views need to be projected. SAM can provide high-quality segmentation results given proper prompts. However, manually selecting prompts from every view is time-consuming and impractical. We propose a cross-view self-prompting mechanism to produce prompts for different novel views automatically.

Specifically, we first render a novel-view 2D segmentation mask M ( n ) from the 3D mask grids V according to Equation (3). This mask is usually inaccurate, especially at the preliminary iteration of SA3D. Then we obtain some point prompts from the rendered mask with a specific strategy. The above process is named cross-view self-prompting. While there are multiple possible solutions for this strategy, we present a feasible one that has been demonstrated to be effective.

Self-prompting Strategy Given an inaccurate 2D rendered mask M ( n ) , the self-prompting strategy aims to extract a set of prompt points P s from it, which can help SAM to generate 2D segmentation result as accurate as possible. It is important to note that M ( n ) is not a typical 2D bitmap, but rather a confidence score map computed using Equation (3). Since each image pixel p corresponds to a ray r in a rendered view, we use p for an easier demonstration of the prompt selection strategy on images.

As P s is initialized to an empty set, the first prompt point p 0 is selected as the point with the highest mask confidence score: p 0 = arg max p M ( n ) ( p ) . To select new prompt points, we first mask out square shaped regions 2 on M ( n ) centered with each existing point prompt ˆ p ∈ P s . Considering the depth z ( p ) can be estimated by the pre-trained NeRF, we transform the 2D pixel p to the 3D point G ( p ) = ( x ( G ( p )) , y ( G ( p )) , z ( G ( p ))) :

where x ( p ) , y ( p ) denote the 2D coordinates of p , and K denotes the camera intrinsics. The new prompt point is expected to have a high confidence score while being close to existing prompt points. Considering the two factors, we introduce a decay term to the confidence score. Let d ( · , · ) denote the min-max normalized Euclidean distance. For each remaining point p in M ( n ) , the decay term is

Then a decayed mask confidence score ˜ M ( n ) ( p ) is computed as

The remaining point with the highest decayed score, i.e. , p ∗ = arg max p / ∈P s ˜ M ( n ) ( p ) , is added to the prompt set: P s = P s ∪ { p ∗ } . The above selection process is repeated until either the number of prompts |P s | reaches a predefined threshold n p or the maximum value of ˜ M ( n ) ( p ) is smaller than 0.

IoU-aware View Rejection When the target object is rendered in heavily occluded views, SAM may produce incorrect segmentation results and degrades the quality of the 3D mask. To avoid such situations, we introduce an additional view rejection mechanism based on the intersection-over-union (IoU) between the rendered mask M ( n ) and the SAM prediction M ( n ) SAM . If the IoU falls below a predefined threshold τ , it indicates a poor overlap between the two masks. The prediction from SAM is rejected, and the mask inverse rendering step is skipped in this iteration.

4 Experiments

In this section, we quantitatively evaluate the segmentation ability of SA3D on various datasets. Then, we qualitatively demonstrate the versatility of SA3D, which can conduct instance segmentation, part segmentation, and text-prompted segmentation etc .

4.1 Datasets

For quantitative experiments, we use the Neural Volumetric Object Selection (NVOS) [47], SPInNeRF [39], and Replica [51] datasets. The NVOS [47] dataset is based on the LLFF dataset [37], which includes several forward-facing scenes. For each scene, NVOS provides a reference view

with scribbles and a target view with 2D segmentation masks annotated. Similar to NVOS, SPInNeRF [39] annotates some data manually to evaluate interactive 3D segmentation performance. These annotations are based on some widely-used NeRF datasets [37, 38, 29, 26, 13]. The Replica [51] dataset provides high-quality reconstruction ground truths of various indoor scenes, including clean dense geometry, high-resolution and high-dynamic-range textures, glass and mirror surface information, semantic classes, planar segmentation, and instance segmentation masks. For qualitative analysis, we use the LLFF [37] dataset and the 360 ◦ dataset [1]. SA3D is further applied to the LERF [23] dataset, which contains more realistic and challenging scenes.

4.2 Quantitative Results

NVOS Dataset For fair comparisons, we follow the experimental setting of the original NVOS [47]. We first scribble on the reference view (provided by the NVOS dataset) to conduct 3D segmentation, and then render the 3D segmentation result on the target view and evaluate the IoU and pixel-wise accuracy with the provided ground truth. Note that the scribbles are preprocessed to meet the requirements of SAM. More details can be found in the supplement.

As shown in Table 1, SA3D outperforms previous approaches by large margins, i.e. , +6.5 mIoU over the previous SOTA ISRF and +20.2 mIoU over the NVOS.

SPIn-NeRF Dataset We follow SPIn-NeRF [39] to conduct label propagation for evaluation. Given a specific reference view of a target object, the 2D ground-truth mask of this view is available. The prompt input operation is omitted, while the 2D ground-truth mask of the target object from the reference view is directly used for the initialization of the 3D mask grids. This is reasonable since in most situations users can refine their input prompts to help SAM generate a 2D mask as accurately as possible from the reference view. Once the 3D mask grids are initialized, the subsequent steps are exactly the same as described in Section 3. With the 3D mask grids finalized, the 2D masks in other views are rendered to calculate the IoU with the 2D ground-truth masks. Results can be found in Table 2. SA3D is demonstrated to be superior in both forward-facing and 360 ◦ scenes.

In Tables 2 and 3, 'Single view' refers to conducting mask inverse rendering exclusively for the 2D ground-truth mask of the reference view. This process is equivalent to mapping the 2D mask to the 3D space based on the corresponding depth information, without any subsequent learnable/updating step. We present these results to demonstrate the gain of the alternated process in our framework. As shown in Table 2, SA3D outperforms MVSeg [39] in most scenes, especially +5.6 mIoU on Truck and +17.3 mIoU on Lego. Besides, compared with the 'Single view' model, a significant promotion is achieved, i.e. +17.8 mIoU, which further proves the effectiveness of our method.

Text:

'The table'

Text:

'The vase'

Text:

'The bonsai'

Replica Dataset We use the processed Replica data with 2D instance labels provided by Zhi et al. [73] to evaluate the segmentation performance of SA3D. We retrieve all views containing each object and specify one reference view. With a similar setting of experiments on the SPIn-NeRF dataset, we use the ground-truth mask of the reference view and perform SA3D to conduct label propagation for evaluation. For each scene in Replica, around 20 objects are chosen for evaluation. Refer to the supplement for more details. As shown in Table 3, the mean IoU (mIoU) is reported for all available objects in different scenes. We exclude the pixel-wise accuracy metric since an object only appears in a few views in the indoor scenes of Replica, where the pixel-wise accuracy is too high to serve as a reliable metric.

In complex indoor scenes of Replica, MVSeg's strategy based on video segmentation proves to be ineffective, which generates numerous inaccurate 2D pseudo-labels, even using the Semantic-NeRF [73] for refinement. Consequently, the final 3D segmentation results of MVSeg even underperform those achieved by the 'Single view' method. In contrast, SA3D accurately captures segmented objects in complex 3D scenes. Visualization results are shown in Figure 3.

4.3 Qualitative Results

We conduct three kinds of segmentation tasks: object segmentation, part segmentation and textprompting segmentation. The first two are the core functions of SA3D. As shown in Figure 3, SA3D demonstrates its capability to segment diverse 3D objects across different scenes, even when the objects are of small scales. Besides, SA3D can also handle challenging part segmentation. The last row of the figure showcases SA3D's precise segmentation of the bucket, small wheel, and dome light of the Lego bulldozer. Figure 4 demonstrates the potential of SA3D in combining with language models. Given a text phrase, the corresponding object can be accurately cut out. The text-prompting segmentation is built upon Grounding-DINO [33], a model capable of generating bounding boxes for objects based on text prompts. These bounding boxes serve as input prompts for SA3D in the segmentation process.

4.4 Ablation Study

Number of Views The process of mask inverse rendering and cross-view self-prompting is alternated across different views. By default, we utilize all available views in the training set I . However, to expedite the 3D segmentation procedure, the number of views can be reduced. As shown in Table 4, We perform experiments on two representative scenes from the SPIn-NeRF [39] dataset to demonstrate this characteristic. The views are uniformly sampled from the sorted training set. In forward facing scenes where the range of the camera poses is limited, satisfactory results can be achieved by selecting only a few views. On an Nvidia RTX 3090 GPU, the 3D segmentation process with 5 views can be completed within 10 seconds. On the contrary, in scenes where the range of camera poses is wider, a larger number of views are required to yield greater improvements. Note that even with 50 views, the segmentation task can still be completed in less than 2 minutes.

Hyper-parameters SA3D involves three hyper-parameters: the IoU rejection threshold τ , the loss balance coefficient λ in Equation (5), and the number of self-prompting points n p . As shown in Table 6, too small τ values lead to unstable SAM predictions, introducing noises to the 3D mask; too large τ values impede the 3D mask from getting substantial information. Table 7 indicates slightly introducing a negative term with the λ factor can reduce noise for mask projection. However, a too-large negative term may make the mask completion process unstable and causes degraded performance. The selection of n p depends on the specific segmentation target, as SAM tends to produce over-segmented results that capture finer details of objects. As shown in Figure 5, for objects with a relatively large scale and complex structures, a bigger n p produces better results. Empirically. setting n p to 3 can meet the requirements of most situations.

Self-prompting Strategy Without the 3D distance based confidence decay (Equation (7)), our self-prompting strategy degrades to a simple 2D NMS (Non-Maximum Suppression), which selects a prompt point with the highest confidence score and then masks out a region around it. To showcase the efficacy of our design, we conduct experiments using the NVOS benchmark and presented per-scene results for in-depth analysis.

Table 5 shows that a simple NMS self-prompting is enough for most cases. But for hard cases like 'LLFF-trex' (a trex skeleton, as shown in Figure 5), where a large number of depth jumps, the confidence decay term contributes a lot. In such a situation, inaccurate masks bleed through gaps in the foreground onto the background. If the self-prompting mechanism generates prompts on these inaccurate regions, SAM may produce plausible segmentation results that can cheat the IoU-rejection mechanism and finally the segmentation results will involve unwanted background regions.

2D Segmentation Models In addition to SAM, we also incorporate four other prompt-based 2D segmentation models [75, 49, 32, 5] into our framework to demonstrate the generalization ability of SA3D. The evaluation results on the NVOS dataset is shown in Table 8.

5 Discussion

On top of the experimental results, we hope to deliver some insights from our preliminary study of integrating SAM and NeRF, i.e. , a 2D foundation model and a 3D representation model.

10 self-prompting points

6 self-prompting points

SAM prediction

SA3D prediction

First, NeRF improves the segmentation quality of SAM. In Figure 6, we show that SA3D can eliminate segmentation errors of SAM and effectively capture details such as holes and edges. Perceptually, SAM, as well as other 2D perception models, is often sensitive to the viewpoint, and NeRF offers the ability of 3D modeling and hence complementariness to assist recognition. Additionally, SA3D inspires us that using NeRF or other 3D structural priors is a resource-efficient method to lift a vision foundation model from 2D to 3D, as long as the foundation model has the ability to self-prompt. This methodology can save many resources because collecting a large corpus of 3D data is often costly. We look forward to research efforts to enhance the 3D perception ability of 2D foundation models ( e.g. , injecting a 3D-aware loss into 2D pre-training).

Limitation SA3D has limitations in panoptic segmentation. First, the current paradigm relies on the first-view prompt. If some objects do not appear in the view for prompting, they will be omitted in the subsequent segmentation process. Second, the same part in the scene may be segmented into different instances with similar semantics in different views. This ambiguity cannot be easily eliminated under the current mechanism design and lead to unstable training. We leave these issues as future work.

6 Conclusion

In this paper, we propose SA3D, a novel framework that generalizes SAM to segment 3D objects with neural radiance fields (NeRFs) as the structural prior. Based on a trained NeRF and a set of prompts in a single view, SA3D performs an iterative procedure that involves rendering novel 2D views, self-prompting SAM for 2D segmentation, and projecting the segmentation back onto 3D mask grids. SA3D can be efficiently applied to a wide range of 3D segmentation tasks. Our research sheds light on a resource-efficient methodology that lifts vision foundation models from 2D to 3D.

References

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR , 2022.

Wang Bing, Lu Chen, and Bo Yang. Dm-nerf: 3d scene geometry decomposition and manipulation from 2d images. arXiv preprint arXiv:2208.07227 , 2022.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV , 2022.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. , 2018.

Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. Focalclick: Towards practical interactive image segmentation. In CVPR , 2022.

Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR , 2022.

Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS , 2021.

Hang Chu, Wei-Chiu Ma, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler. Surfconv: Bridging 3d and 2d convolution for rgbd images. In CVPR , 2018.

Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR , 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776 , 2022.

Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers , 2022.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR , 2022.

Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In 3DV , 2022.

Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and PJ Narayanan. Interactive segmentation of radiance fields. arXiv preprint arXiv:2212.13545 , 2022.

Nikhil Gosala and Abhinav Valada. Bird's-eye-view panoptic segmentation using monocular frontal view images. IEEE Robot. Autom. Lett. , 2022.

Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. In CoRL , 2022.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV , 2017.

Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul E. Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV , 2021.

Benran Hu, Junkai Huang, Yichen Liu, Yu-Wing Tai, and Chi-Keung Tang. Instance neural radiance field. arXiv preprint arXiv:2304.04395 , 2023.

Jing Huang and Suya You. Point cloud labeling using 3d convolutional neural network. In ICPR , 2016.

Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241 , 2023.

Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553 , 2023.

Alexander Kirillov, Kaiming He, Ross B. Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR , 2019.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans. Graph. , 2017.

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In NeurIPS , 2022.

Ruofan Liang, Jiahao Zhang, Haoda Li, Chen Yang, and Nandita Vijaykumar. Spidr: Sdf-based neural point fields for illumination and deformation. arXiv preprint arXiv:2210.08398 , 2022.

Yen-Chen Lin, Pete Florence, Jonathan T. Barron, Tsung-Yi Lin, Alberto Rodriguez, and Phillip Isola. Nerf-supervision: Learning dense object descriptors from neural radiance fields. In ICRA , 2022.

David B. Lindell, Julien N. P. Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In CVPR , 2021.

Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su. Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models. In CVPR , 2023.

Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. In ICCV , 2023.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

Xinhang Liu, Jiaben Chen, Huai Yu, Yu-Wing Tai, and Chi-Keung Tang. Unsupervised multiview object segmentation using radiance field propagation. In NeurIPS , 2022.

Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. NeurIPS , 2019.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , 2015.

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. , 2019.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Ashkan Mirzaei, Tristan Aumentado-Armstrong, Konstantinos G. Derpanis, Jonathan Kelly, Marcus A. Brubaker, Igor Gilitschenski, and Alex Levinshtein. SPIn-NeRF: Multiview segmentation and perceptual inpainting with neural radiance fields. In CVPR , 2023.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. , 2022.

Michael Niemeyer and Andreas Geiger. GIRAFFE: representing scenes as compositional generative neural feature fields. In CVPR , 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019.

Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR , 2023.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS , 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Zhongzheng Ren, Aseem Agarwala, Bryan C. Russell, Alexander G. Schwing, and Oliver Wang. Neural volumetric object selection. In CVPR , 2022.

Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut": interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. , 2004.

Konstantin Sofiiuk, Ilya A Petrov, and Anton Konushin. Reviving iterative training with mask guidance for interactive segmentation. In ICIP , 2022.

Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01148 , 2021.

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 , 2019.

Robin Strudel, Ricardo Garcia Pinel, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV , 2021.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR , 2022.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprint arXiv:2212.13545 , 2022.

Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In ECCV , 2020.

Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng. Delicate textured mesh recovery from nerf via adaptive surface refinement. arXiv preprint arXiv:2303.02091 , 2023.

Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. In 3DV , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. arXiv preprint arXiv:2111.13260 , 2021.

Weiyue Wang and Ulrich Neumann. Depth-aware cnn for rgb-d segmentation. In ECCV , 2018.

Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. Nex: Real-time view synthesis with neural basis expansion. In CVPR , 2021.

Zongwei Wu, Guillaume Allibert, Christophe Stolz, Chao Ma, and Cédric Demonceaux. Depthadapted cnns for rgb-d semantic segmentation. arXiv preprint arXiv:2206.03939 , 2022.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS , 2021.

Yajie Xing, Jingbo Wang, and Gang Zeng. Malleable 2.5 d convolution: Learning receptive fields along the depth-axis for rgb-d scene parsing. In ECCV , 2020.

Jihan Yang, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. arXiv preprint arXiv:2304.00962 , 2023.

Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T Barron, and Ben Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv preprint arXiv:2302.14859 , 2023.

Dongqiangzi Ye, Zixiang Zhou, Weijia Chen, Yufei Xie, Yu Wang, Panqu Wang, and Hassan Foroosh. Lidarmultinet: Towards a unified multi-task network for lidar perception. arXiv preprint arXiv:2209.09385 , 2022.

Hong-Xing Yu, Leonidas J. Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In ICLR , 2022.

Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. arXiv preprint arXiv:2303.04748 , 2023.

Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In ICCV , 2021.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR , 2017.

Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR , 2021.

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV , 2021.

Hui Zhou, Xinge Zhu, Xiao Song, Yuexin Ma, Zhe Wang, Hongsheng Li, and Dahua Lin. Cylinder3d: An effective 3d framework for driving-scene lidar semantic segmentation. arXiv preprint arXiv:2008.01550 , 2020.

Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718 , 2023.

Appendix

The appendix includes the following contents:

Implementation details (Section A);

A segmentation refinement strategy (Section B);

The scribble to point strategy for the evaluation of NVOS (Section C);

An analysis about vanilla NeRF [38] used in SA3D (Section D);

More information about the object filtering of Replica [51] (Section E);

A further illustration of the self-prompting strategy (Section F);

An analysis about the effect of different kinds of occlusion in NeRF (Section G);

More visualization results with different kinds of input prompts (Section H).

A Implementation Details

We implement SA3D using PyTorch [42] with reference to the code provided by DVGOv2 [54]. The SA3D model is built and trained on a single Nvidia Geforce RTX3090 GPU. For our NeRF model, we primarily employ TensoRF [3], utilizing the VM-48 representation to store the radiance latent vectors. The radiance fields are pre-trained for most datasets with 40,000 iterations. For the LLFF dataset [37] and the 360 dataset [1], the radiance fields are trained with 20,000 iterations.

B Refinement with A Two-pass Segmentation Mechanism

SAM may produce segmentation masks containing undesired parts. The IoU-aware view rejection is hard to handle this issue when the mis-classified region gradually expands.

We propose a two-pass segmentation mechanism to further refine the segmentation result. After completing 3D segmentation introduced in the main manuscript, we get a 3D mask V . To detect the mis-classified region from V , we re-render the 2D segmentation mask M u of the user-specific reference view and compare it with the original SAM segmentation result M u SAM .

Subsequently, we reset the original 3D mask V to be a zero tensor and introduce another 3D mask V ′ ∈ R 3 that specifically indicates the mis-classified regions. The 3D segmentation process is then repeated, with the key difference being the incorporation of negative prompt points during the self-prompting phase. In other words, the prompts obtained from V ′ serve as negative prompts for V , and vice versa. This incorporation of negative prompts enables SAM to gain a better understanding of the user's requirements and refine the segmentation accordingly (shown in Figure B7). It is important to note that while this two-pass segmentation mechanism holds promise, it was not utilized in our main experiments due to considerations of efficiency.

Vanilla SA3D

C The Scribble to Points Strategy for The Evaluation of NVOS

The NVOS [47] dataset provides a reference view and the corresponding scribbles for each scene (shown in Figure C8). In practice, since the scribbles usually contain tens of thousands of dense

影

points, SAM [25] cannot directy take such scribbles as input. The abundance of points in the scribbles hinders SAM's performance when directly utilizing them as prompts, which is an inherent limitation of SAM.

For fair comparison, we extract positive and negative prompt points from the provided positive and negative scribbles, respectively. For input scribbles, we first skeletonize them and then select 2% points from the skeletonized positive scribbles as the positive prompts and 0 . 5% points from the skeletonized negative scribbles as the negative prompts.

D The Effect of Different NeRFs Used in SA3D

We adapt SA3D to the vanilla NeRF [38] to showcase its generalizability. We present visualization results on the LLFF dataset. As illustrated in Figure D9, SA3D with the vanilla NeRF exhibits excellent performance without the need for additional modifications.

E Object Filtering for The Replica Dataset

The Replica dataset contains many objects in each scene. However, it is important to note that many of these objects exhibit low quality, as depicted in Figure E10, making them unsuitable for evaluating 3D segmentation. Generally, these instances exhibit the following issues: some instances are not present in the training frames provided by Zhi et al. [73]; some instances are too small to be effectively segmented, such as thin slits in doors; and some instances consist of unrecognizable, low-resolution pixels, such as blurred tags, which are not suitable for accurate instance segmentation. Accordingly, we carefully select approximately 20 representative instances from each scene for the evaluation. The list of instance IDs for each scene can be found in Table E10. We have also included the quantitative results without object filtering in Table E9. Even without object filtering, SA3D demonstrates improvements compared to the single-view baseline.

F An Illustration for The Proposed Self-prompting Strategy

We offer an illustration (Figure F11) to assist readers in gaining a clearer understanding of the self-prompting strategy.

Flower

Fortress

Fern

Orchids

Trex

F

In the self-prompting strategy, prompt points P s are derived from an incomplete 2D rendered mask M ( n ) , which is represented as a confidence score map. Initially, the selected prompt points set P s is empty, and the first prompt point p 0 is selected as the one with the highest confidence score in the mask M ( n ) . For subsequent points, square regions centered around existing prompt points are masked out on M ( n ) . The depth z ( p ) , estimated by the pre-trained NeRF, helps convert 2D pixel p into a 3D point G ( p ) . The new prompt point is expected to have a high confidence score while being close to existing prompt points. Hence, a distance-aware decay term is introduced to compute the confidence score. The remaining point with the highest decayed score is added to the prompt set. This selection process is repeated until either the number of prompts |P s | reaches a predefined threshold n p or the maximum value of the remaining points is less than 0. Please refer to Section 3.4 of the main manuscript for more details.

G The Effect of Occlusion in NeRF

There are two cases of occlusion in NeRFs: part of the target object does not appear in some views (but appear in other views), or it does not appear at all. For the former case, SA3D can recover it using information from other views; for the latter case, there can be parts missing in the target object (see Figure G12). This is an interesting future direction ( e.g. , applying generative models such as diffusion models).

H More Visualization Results

Wepresent additional visualization results in Figure H13 and Figure H15, showcasing the effectiveness of SA3D across various input prompts. We also provide some visualization of extracted meshes of the segmented objects (Figure H14) to show the extracted 3D geometry. Please note that the quality of these meshes can be further improved by applying more effective NeRF2Mesh methods [56, 66].

2

,

3D distance-based

confidence decay

3

,

Select  a new point with the highest confidence

1

,

Mask out a square region

Mask out square regions

Equation (6),(7),(8)

.

Occluded but seen in other views

Never seen in any views

'The sheep'

'The paper'

'The apple'

'The cup'

'The bag'

'The dish'

BALI-S

PALL-E

Forward facing

Chalkboard

小

Fern

Shoerack

360°

Kitchen

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a simple and efficient method for general 3D segmentation based on SAM (a powerful 2D foundation model) and NeRF-style representation. Instead of building a 3D foundation model from scratch, SA3D uses two steps to lifts 2D SAM segmentation results to 3D in a more concise and efficient way. Based on a well-trained NeRF, a rendered reference view and the human input prompts are sent into SAM to get the first segmentation. Then the proposed mask inverse rendering and cross-view self-prompting strategy will help optimize a volume-based 3D mask field and propagate segmentation information to different views in an iterative and incremental manner. The comprehensive experiments prove the effectiveness of the designed pipeline.\n\nSTRENGTHS:\n1.\tThe proposed method is simple but effective. It provides a general interactive 3D object segmentation paradigm which do not rely on heavy pre-training.\n2.\tThe method is pretty efficient. Given a pre-trained NeRF, the 3D segmentation can be completed within only minutes with the help of SAM.\n3.\tThe experiments have covered variant datasets and comparison to SOTA methods, which are comprehensive and persuasive.\n4.\tThe paper is well-written and easy to follow.\n\nWEAKNESSES:\n1.\tI expect that segment “anything” in 3D means the method can segment all the things in a 3D scene, so that it is consistent with the purpose of SAM. But the proposed SA3D can only segment one object at each time.\n2.\tFurthermore, using prompts (such as points, scribbles, text) to achieve 2D segmentation of one target object is a long-studied subject. Besides SAM, there should be many alternative choices such as [1][2][3], which should have been evaluated for this 3D task. Or the irreplaceability of SAM needs to be explained.\n3.\tI wondering if the method requires complete observation of the target object in at least one image. What if the target object is not observed completely in any views? How will the choice of reference view affect the 3D segment results? Is there any necessary strategy to achieve best performance? The robustness of the method should be evaluated.\n\n[1] Sofiiuk K, Petrov I A, Konushin A. Reviving iterative training with mask guidance for interactive segmentation[C]//2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2022: 3141-3145.\n[2] Liu Q, Xu Z, Bertasius G, et al. SimpleClick: Interactive image segmentation with simple vision transformers[J]. arXiv preprint arXiv:2210.11006, 2022.\n[3] Chen X, Zhao Z, Zhang Y, et al. Focalclick: Towards practical interactive image segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 1300-1309.\n\nQUESTIONS:\n1.\tIt seems that the method is not restricted by NeRF-based representation. Since the authors assume NeRF provides good depth, the proposed method should also work under other representations such as mesh and point cloud?\n2.\tThe 3D mask is modeled as dense voxel grid, which may cause high memory consumption for high resolution representation. How will different resolutions affect the performance? Maybe using a MLP is a better choice?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to lift 2D segmentations from foundation models such as SAM to 3D by iterating between SAM and NeRF, without re-training or re-defining either. Given a trained NeRF model, the model first renders a view, which is also processed by SAM given a user click. With the segmentation by SAM, the model optimizes a 3D segmentation volume such that it volume renders into a mask consistent with what SAM produced (“mask inverse rendering”). Next, the model projects this initial segmentation volume to other viewpoints, producing incomplete 2D masks. Finally, the model computes “good prompts” for SAM to complete these masks (“cross-view self prompting”). The model iterates between these steps until a complete segmentation volume has been produced.\n\nBecause of the generalization power of SAM, the model is able to segment almost anything in 3D, without requiring applying changes to SAM or NeRF, making itself a framework that can be applied to any 2D foundation models that we want to lift to 3D.\n\nSTRENGTHS:\nThe proposed method is simple yet effective, following the recent trend of developing foundation models and/or open-vocabulary LLMs. More importantly, it also bridges the gap between powerful 2D models and 3D understanding as required by robots or autonomous vehicles. It is general and applicable to any 2D foundation models. The results are strong both qualitatively and quantitatively.\n\nWEAKNESSES:\nSince the framework is claimed to be (and I think it is) general and applicable to any 2D foundation models, the paper will be much stronger if the authors could demonstrate the use of this framework to lift another foundation model’s output into 3D. \n\nThe paper will also benefit from showing some 3D shape results that are extracted from the 3D segmentation volume. Often, it’s the underlying 3D geometry that matters for, say, robotic manipulation. “RGB looking good” is separated than that.\n\nQUESTIONS:\nHas runtime been reported? I recommend augmenting Table 4 with an additional row of runtime.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for segmenting a pre-trained NeRF by utilizing SAM. Given a pre-trained NeRF, it first asks users to provide prompts (e.g., some points) for a reference view. It then utilizes the SAM to generate a 2D segmentation for the reference view and utilizes the 2D mask to optimize a 3D segmentation mask. After that, it will iteratively update the 3D mask by (a) selecting a random review, (b) rendering the view, (c) utilizing the 3D mask to automatically generate prompts for the view, (d) utilizing prompts to query SAM, (e) utilizing SAM output to update 3D mask. The method proposes a self-promoting strategy to generate prompts given the 3D mask and an IoU-aware view rejection to ignore the bad prediction from SAM.\n\nSTRENGTHS:\n1. The paper proposes a novel method for segmenting a pre-trained NeRF by using SAM.\n2. The paper proposes a self-promoting strategy for automatically generating prompts for SAM and a rejection strategy to ignore bad SAM predictions.\n3. The authors conduct experiments on three datasets and provide some ablation studies.\n4. The paper is easy to follow.\n\nWEAKNESSES:\n1. The method requires minutes for a single segmentation (e.g., of an object), which may greatly limit the usage of the method in many real-world applications (e.g., robotics manipulation).\n\n2. Recently, there are also many NeRF/Point Cloud-based open-world (vocabulary) 3D segmentation (for both scene-level and part-level) methods[1-8] that leverage pre-trained 2D VLM (e.g., CLIP). However, the discussion and comparison with them is missing. It seems that many of these prior methods don't need per-instance optimization and can generate a 3D segmentation mask in just seconds. Please cite these papers and discuss the advantages of the proposed methods.\n\n3. In Line 153, the paper states, \"Given an incomplete 2D rendered mask\". Why is the rendered mask always incomplete? Is it possible that some SAM predictions are wrong and include extra regions, which leads to an enlarged 3D segmentation mask? If this is possible, the self-prompting strategy will also generate a wrong prompt for SAM in the later steps. Please explain whether this case is possible and how the proposed method can handle the wrong SAM prediction (including extra regions).\n\n4. The negative refinement term (Line 137) is unclear to me. Please explain in more detail about the motivation for this term.\n\n5. Equation (7) is not clear to me. Could you explain in more detail? Also, it would be better to provide an ablation study to verify the necessity of the confidence decay step (Equation (7)). Can the self-prompting strategy still work without confidence decay?\n\n6. For all tables, could you include runtime for both the proposed method and baseline methods?\n\n7. It would be better to include evaluations on some standard 3D segmentation benchmarks (e.g., ScanNet and PartNet) to have extensive comparison with existing methods as well. \n\n\n\n[1] Kerr, Justin, et al. \"Lerf: Language embedded radiance fields.\" arXiv preprint arXiv:2303.09553 (2023).\n\n[2] Peng, Songyou, et al. \"Openscene: 3d scene understanding with open vocabularies.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Ding, Runyu, et al. \"PLA: Language-Driven Open-Vocabulary 3D Scene Understanding.\" arXiv preprint arXiv:2211.16312 (2022).\n\n[4] Ha, Huy, and Shuran Song. \"Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models.\" 6th \nAnnual Conference on Robot Learning. 2022.\n\n[5] Zhang, Junbo, Runpei Dong, and Kaisheng Ma. \"Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip.\" arXiv preprint arXiv:2303.04748 (2023).\n\n[6] Liu, Minghua, et al. \"Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[7] Yang, Jihan, et al. \"Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding.\" arXiv preprint arXiv:2304.00962 (2023).\n\n[8] Jatavallabhula, Krishna Murthy, et al. \"Conceptfusion: Open-set multimodal 3d mapping.\" arXiv preprint arXiv:2302.07241 (2023).\n\nQUESTIONS:\n1. How many gradient descent iterations do you need in each inverse rendering step?\n\n2. In Table 2, SA3D performs worse than the baseline method (MVSeg). Is there any common pattern for these instances? Or any reasons for the performance differences?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a combination of the newly introduced Segment-Anything Model (SAM) with Nerf, yielding the  Segment Anything in 3D (SA3D) system.\nSA3D cam take a 3D scene reconstructed by Nerf and based on a user prompt (e.g. a few keypoints) can carve out distinct 3D objects from the scene. This is shown to outperform previous state-of-the-art systems. \nThey two novel bits to successfully combine the two approaches:\nFirstly a loss function in order to induce a 3D mask field on a voxel grid based on the SAM outputs (Eq. 4) and secondly (and most importantly) a method to prompt SAM on another view based on the existing 3D mask view - allowing one to only prompt from one image and then perform \"prompt propagation\" to the rest.\n\nSTRENGTHS:\nSimple approach that should be easy to reproduce - the authors also share their code, which looks reasonably clean. \n\nSolid experimental improvements - I am convinced by these numbers that the method outperforms some of the latest state-of-the-art systes.\n\nGood ablations.\n\nInteresting -but a bit hand-wavy - results that SA3D can improve SAM; some more validation of this would be welcome.\n\nWEAKNESSES:\n- Overstatement: the authors are  using SAM in tandem with Nerf, to get 3D objects out of a scene, which is great - but practically this is similar to classic co-segmentation or the systems that they compare to, but a bit better because of piggy-backing on SAM. \nStating in the abstract that  \"Our research offers a generic and efficient methodology to lift a 2D vision foundation model to 3D, as long as the 2D model can steadily address  promptable segmentation across multiple views.\" suggests more than just this - one can start imagining extending the attention operations to 3D, supervising for depth/volumetric reconstruction etc, or more importantly having a foundation model for 3D, none of which is  the case based on what we have in the present paper. Technically the statement is not false - but implies more than what is actually happenning in the paper. \n\n\n- I could not find any discussion about the computational efficiency of the proposed algorithm. At the moment it's unclear what is the importance of this.\n\nMinor:\n\n- Unclear intuitively what the second term does in Eq. 5 from the way it's written. I think a more obvious rewrite is \n(lambda - 1) L_proj + lambda sum_r M(r) \\propto L_proj + lambda/(lambda-1) sum_r M(r), \nand explain that the second term is just a regularization term on the optimized segmentation field.\n\n- l. 280: \"we demonstrate limitations of SA3D in panoptic segmentation\" -> could not find any pointer to this in the paper.\n\nQUESTIONS:\n-some more validation of the statement that SA3D can improve SAM would be welcome. \n\n- Self prompting strategy: I am not too sure I see the necessity of this particular prompting strategy. The authors are lifting every prompt point to 3D, reducing the mask score in 3D in its neighborhood, regenerating the mask, and picking the next strongest point. How much better is this than a plain 2D-based nonmaximum suppression strategy? This should be  more efficient computationally and yield roughly the same results - if not, how much worse are they?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}