SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SPIn-NeRF Dataset We follow SPIn-NeRF [39] to conduct label propagation for evaluation. Given a specific reference view of a target object, the 2D ground-truth mask of this view is available. The prompt input operation is omitted, while the 2D ground-truth mask of the target object from the reference view is directly used for the initialization of the 3D mask grids. This is reasonable since in most situations users can refine their input prompts to help SAM generate a 2D mask as accurately as possible from the reference view. Once the 3D mask grids are initialized, the subsequent steps are exactly the same as described in Section 3. With the 3D mask grids finalized, the 2D masks in other views are rendered to calculate the IoU with the 2D ground-truth masks. Results can be found in Table 2. SA3D is demonstrated to be superior in both forward-facing and  $360^{\circ}$  scenes.
In Tables 2 and 3, "Single view" refers to conducting mask inverse rendering exclusively for the 2D ground-truth mask of the reference view. This process is equivalent to mapping the 2D mask to the 3D space based on the corresponding depth information, without any subsequent learnable/updating step. We present these results to demonstrate the gain of the alternated process in our framework. As shown in Table 2, SA3D outperforms MVSeg [39] in most scenes, especially  $+5.6$  mIoU on Truck and  $+17.3$  mIoU on Lego. Besides, compared with the "Single view" model, a significant promotion is achieved, i.e.  $+17.8$  mIoU, which further proves the effectiveness of our method.
Table 3: Quantitative results on Replica (mIoU).
Scenes	office0	office1	office2	office3	office4	room0	room1	room2	mean
Single view	68.7	56.5	68.4	62.2	57.0	55.4	53.8	56.7	59.8
MVSeg [39]	31.4	40.4	30.4	30.5	25.4	31.1	40.7	29.2	32.4
SA3D (ours)	84.4	77.0	88.9	84.4	82.6	77.6	79.8	89.2	83.0
Figure 4: 3D segmentation results of SA3D with the text prompts in 360-garden [1].
Text: "The table"
Text: "The vase"
Text: "The bonsai"
Replica Dataset We use the processed Replica data with 2D instance labels provided by Zhi et al. [73] to evaluate the segmentation performance of SA3D. We retrieve all views containing each object and specify one reference view. With a similar setting of experiments on the SPIn-NeRF dataset, we use the ground-truth mask of the reference view and perform SA3D to conduct label propagation for evaluation. For each scene in Replica, around 20 objects are chosen for evaluation. Refer to the supplement for more details. As shown in Table 3, the mean IoU (mIoU) is reported for all available objects in different scenes. We exclude the pixel-wise accuracy metric since an object only appears in a few views in the indoor scenes of Replica, where the pixel-wise accuracy is too high to serve as a reliable metric.
In complex indoor scenes of Replica, MVSeg's strategy based on video segmentation proves to be ineffective, which generates numerous inaccurate 2D pseudo-labels, even using the Semantic-NeRF [73] for refinement. Consequently, the final 3D segmentation results of MVSeg even underperform those achieved by the "Single view" method. In contrast, SA3D accurately captures segmented objects in complex 3D scenes. Visualization results are shown in Figure 3.
We conduct three kinds of segmentation tasks: object segmentation, part segmentation and text-prompting segmentation. The first two are the core functions of SA3D. As shown in Figure 3, SA3D demonstrates its capability to segment diverse 3D objects across different scenes, even when the objects are of small scales. Besides, SA3D can also handle challenging part segmentation. The last row of the figure showcases SA3D's precise segmentation of the bucket, small wheel, and dome light of the Lego bulldozer. Figure 4 demonstrates the potential of SA3D in combining with language models. Given a text phrase, the corresponding object can be accurately cut out. The text-prompting segmentation is built upon Grounding-DINO [33], a model capable of generating bounding boxes for objects based on text prompts. These bounding boxes serve as input prompts for SA3D in the segmentation process.
Table 4: Ablation on different numbers of views for 3D mask generation. Numbers in parentheses represent the view percentage of total training views.
Number of Views	5 (10%)	9 (20%)	21 (50%)	43 (100%)
IoU on Fortress (forward facing)	97.8	98.3	98.3	98.3
Time Cost (s)	7.6	12.8	29.0	59.0
Number of Views	11 (10%)	21 (20%)	51 (50%)	103 (100%)
IoU on Lego (360°)	84.5	84.8	91.5	92.2
Time Cost (s)	23.5	43.5	103.8	204.9
Number of Views The process of mask inverse rendering and cross-view self-prompting is alternated across different views. By default, we utilize all available views in the training set  $\mathcal{I}$ . However, to expedite the 3D segmentation procedure, the number of views can be reduced. As shown in Table 4, We perform experiments on two representative scenes from the SPIn-NeRF [39] dataset to demonstrate this characteristic. The views are uniformly sampled from the sorted training set. In forward facing scenes where the range of the camera poses is limited, satisfactory results can be achieved by selecting only a few views. On an Nvidia RTX 3090 GPU, the 3D segmentation process with 5 views can be completed within 10 seconds. On the contrary, in scenes where the range of camera poses is wider, a larger number of views are required to yield greater improvements. Note that even with 50 views, the segmentation task can still be completed in less than 2 minutes.
Hyper-parameters SA3D involves three hyper-parameters: the IoU rejection threshold  $\tau$ , the loss balance coefficient  $\lambda$  in Equation (5), and the number of self-prompting points  $n_p$ . As shown in Table 6, too small  $\tau$  values lead to unstable SAM predictions, introducing noises to the 3D mask; too large  $\tau$  values impede the 3D mask from getting substantial information. Table 7 indicates slightly introducing a negative term with the  $\lambda$  factor can reduce noise for mask projection. However, a too-large negative term may make the mask completion process unstable and causes degraded performance. The selection of  $n_p$  depends on the specific segmentation target, as SAM tends to produce over-segmented results that capture finer details of objects. As shown in Figure 5, for objects with a relatively large scale and complex structures, a bigger  $n_p$  produces better results. Empirically, setting  $n_p$  to 3 can meet the requirements of most situations.
Self-prompting Strategy Without the 3D distance based confidence decay (Equation (7)), our self-prompting strategy degrades to a simple 2D NMS (Non-Maximum Suppression), which selects a prompt point with the highest confidence score and then masks out a region around it. To showcase the efficacy of our design, we conduct experiments using the NVOS benchmark and presented per-scene results for in-depth analysis.
Table 5: Ablation on the confidence decay term of the self-prompting strategy.
Scenes	w/ Confidence Decay Term	w/o Confidence Decay Term
IoU (%)	Acc (%)	IoU (%)	Acc (%)
Fern	82.9	94.4	82.9	94.4
Flower	94.6	98.7	94.6	99.7
Fortress	98.3	99.7	98.4	99.7
Horns (center)	96.2	99.3	96.2	99.3
Horns (Left)	90.2	99.4	88.8	99.3
Leaves	93.2	99.6	93.2	99.6
Orchids	85.5	97.3	85.4	97.3
Trex	82.0	97.4	64.0	93.3
mean	90.3	98.2	87.9	97.7
Table 5 shows that a simple NMS self-prompting is enough for most cases. But for hard cases like 'LLFF-trex' (a trex skeleton, as shown in Figure 5), where a large number of depth jumps, the confidence decay term contributes a lot. In such a situation, inaccurate masks bleed through gaps in the foreground onto the background. If the self-prompting mechanism generates prompts on these inaccurate regions, SAM may produce plausible segmentation results that can cheat the IoU-rejection mechanism and finally the segmentation results will involve unwanted background regions.
2D Segmentation Models In addition to SAM, we also incorporate four other prompt-based 2D segmentation models [75, 49, 32, 5] into our framework to demonstrate the generalization ability of SA3D. The evaluation results on the NVOS dataset is shown in Table 8.
On top of the experimental results, we hope to deliver some insights from our preliminary study of integrating SAM and NeRF, i.e., a 2D foundation model and a 3D representation model.
Table 6: Ablation on different IoU-aware rejection threshold  $\tau$  on the Replica office0.
τ	0.3	0.4	0.5	0.6	0.7
mIoU	74.9	79.7	84.4	81.3	82.0
Table 7: Ablation on different negative term coefficient  $\lambda$  on the Replica office0.
λ	0.05	0.1	0.15	0.3	0.5
mIoU	79.1	82.9	84.4	84.9	83.3
Table 8: Ablation on different 2D segmentation models.
SEEM [75]	SimpleClick [32]	RITM [49]	FocalClick [5]
mIoU (%)	mAcc (%)	mIoU (%)	mAcc (%)	mIoU (%)	mAcc (%)	mIoU (%)	mAcc (%)
86.0	97.0	87.7	97.8	81.2	96.3	88.9	98.1
Input image with prompts
Figure 5: Results of different self-prompting points numbers  $n_p$  on the LLFF-trex [37] scene.
3 self-prompting points
6 self-prompting points
10 self-prompting points
Figure 6: The 2D segmentation result by SAM and the 3D segmentation result by SA3D of the LLFF-fern [37] scene. SA3D produces more details that are missing in the SAM segmentation result.
First, NeRF improves the segmentation quality of SAM. In Figure 6, we show that SA3D can eliminate segmentation errors of SAM and effectively capture details such as holes and edges. Perceptually, SAM, as well as other 2D perception models, is often sensitive to the viewpoint, and NeRF offers the ability of 3D modeling and hence complementariness to assist recognition. Additionally, SA3D inspires us that using NeRF or other 3D structural priors is a resource-efficient method to lift a vision foundation model from 2D to 3D, as long as the foundation model has the ability to self-prompt. This methodology can save many resources because collecting a large corpus of 3D data is often costly. We look forward to research efforts to enhance the 3D perception ability of 2D foundation models (e.g., injecting a 3D-aware loss into 2D pre-training).
**Limitation** SA3D has limitations in panoptic segmentation. First, the current paradigm relies on the first-view prompt. If some objects do not appear in the view for prompting, they will be omitted in the subsequent segmentation process. Second, the same part in the scene may be segmented into different instances with similar semantics in different views. This ambiguity cannot be easily eliminated under the current mechanism design and lead to unstable training. We leave these issues as future work.
In this paper, we propose SA3D, a novel framework that generalizes SAM to segment 3D objects with neural radiance fields (NeRFs) as the structural prior. Based on a trained NeRF and a set of prompts in a single view, SA3D performs an iterative procedure that involves rendering novel 2D views, self-prompting SAM for 2D segmentation, and projecting the segmentation back onto 3D mask grids. SA3D can be efficiently applied to a wide range of 3D segmentation tasks. Our research sheds light on a resource-efficient methodology that lifts vision foundation models from 2D to 3D.
This work was supported by NSFC 62322604, NSFC 62176159, Natural Science Foundation of Shanghai 21ZR1432200, and Shanghai Municipal Science and Technology Major Project 2021SHZDZX0102. Specifically, We extend our sincere thanks to Weichao Qiu for his insightful suggestions during the course of this work.
[1] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[2] Wang Bing, Lu Chen, and Bo Yang. Dm-nerf: 3d scene geometry decomposition and manipulation from 2d images. arXiv preprint arXiv:2208.07227, 2022.
[3] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
[4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell., 2018.
[5] Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. Focalclick: Towards practical interactive image segmentation. In CVPR, 2022.
[6] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
[7] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021.
[8] Hang Chu, Wei-Chiu Ma, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler. Surfconv: Bridging 3d and 2d convolution for rgb images. In CVPR, 2018.
[9] Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR, 2023.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[11] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776, 2022.
[12] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, 2022.
[13] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[14] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In 3DV, 2022.
[15] Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and PJ Narayanan. Interactive segmentation of radiance fields. arXiv preprint arXiv:2212.13545, 2022.
[16] Nikhil Gosala and Abhinav Valada. Bird's-eye-view panoptic segmentation using monocular frontal view images. IEEE Robot. Autom. Lett., 2022.
The appendix includes the following contents:
1. Implementation details (Section A);
2. A segmentation refinement strategy (Section B);
3. The scribble to point strategy for the evaluation of NVOS (Section C);
4. An analysis about vanilla NeRF [38] used in SA3D (Section D);
5. More information about the object filtering of Replica [51] (Section E);
6. A further illustration of the self-prompting strategy (Section F);
7. An analysis about the effect of different kinds of occlusion in NeRF (Section G);
8. More visualization results with different kinds of input prompts (Section H).
We implement SA3D using PyTorch [42] with reference to the code provided by DVGOv2 [54]. The SA3D model is built and trained on a single Nvidia Geforce RTX3090 GPU. For our NeRF model, we primarily employ TensoRF [3], utilizing the VM-48 representation to store the radiance latent vectors. The radiance fields are pre-trained for most datasets with 40,000 iterations. For the LLFF dataset [37] and the 360 dataset [1], the radiance fields are trained with 20,000 iterations.
B Refinement with A Two-pass Segmentation Mechanism
SAM may produce segmentation masks containing undesired parts. The IoU-aware view rejection is hard to handle this issue when the mis-classified region gradually expands.
We propose a two-pass segmentation mechanism to further refine the segmentation result. After completing 3D segmentation introduced in the main manuscript, we get a 3D mask  $\mathbf{V}$ . To detect the mis-classified region from  $\mathbf{V}$ , we re-render the 2D segmentation mask  $\mathbf{M}^u$  of the user-specific reference view and compare it with the original SAM segmentation result  $\mathbf{M}_{\mathrm{SAM}}^u$ .
Subsequently, we reset the original 3D mask  $\mathbf{V}$  to be a zero tensor and introduce another 3D mask  $\mathbf{V}' \in \mathbb{R}^3$  that specifically indicates the mis-classified regions. The 3D segmentation process is then repeated, with the key difference being the incorporation of negative prompt points during the self-prompting phase. In other words, the prompts obtained from  $\mathbf{V}'$  serve as negative prompts for  $\mathbf{V}$ , and vice versa. This incorporation of negative prompts enables SAM to gain a better understanding of the user's requirements and refine the segmentation accordingly (shown in Figure B7). It is important to note that while this two-pass segmentation mechanism holds promise, it was not utilized in our main experiments due to considerations of efficiency.
Vanilla SA3D
Two-pass Segmentation
Figure B7: The effect of the two-pass segmentation refinement.
C The Scribble to Points Strategy for The Evaluation of NVOS
The NVOS [47] dataset provides a reference view and the corresponding scribbles for each scene (shown in Figure C8). In practice, since the scribbles usually contain tens of thousands of dense
Figure C8: Some scribbles on the reference views provided by NVOS and the corresponding segmentation results of SA3D. Blue scribbles are positive and red scribbles are negative.
points, SAM [25] cannot directly take such scribbles as input. The abundance of points in the scribbles hinders SAM's performance when directly utilizing them as prompts, which is an inherent limitation of SAM.
For fair comparison, we extract positive and negative prompt points from the provided positive and negative scribbles, respectively. For input scribbles, we first skeletonize them and then select  $2\%$  points from the skeletonized positive scribbles as the positive prompts and  $0.5\%$  points from the skeletonized negative scribbles as the negative prompts.
D The Effect of Different NeRFs Used in SA3D
We adapt SA3D to the vanilla NeRF [38] to showcase its generalizability. We present visualization results on the LLFF dataset. As illustrated in Figure D9, SA3D with the vanilla NeRF exhibits excellent performance without the need for additional modifications.
E Object Filtering for The Replica Dataset
The Replica dataset contains many objects in each scene. However, it is important to note that many of these objects exhibit low quality, as depicted in Figure E10, making them unsuitable for evaluating 3D segmentation. Generally, these instances exhibit the following issues: some instances are not present in the training frames provided by Zhi et al. [73]; some instances are too small to be effectively segmented, such as thin slits in doors; and some instances consist of unrecognizable, low-resolution pixels, such as blurred tags, which are not suitable for accurate instance segmentation. Accordingly, we carefully select approximately 20 representative instances from each scene for the evaluation. The list of instance IDs for each scene can be found in Table E10. We have also included the quantitative results without object filtering in Table E9. Even without object filtering, SA3D demonstrates improvements compared to the single-view baseline.
Table E9: Quantitative results on Replica (mIoU) without object filtering.
Scenes	office0	office1	office2	office3	office4	room0	room1	room2	mean
Single view	58.8	53.1	61.2	54.2	56.7	51.0	58.6	58.3	56.49
SA3D (ours)	65.1	59.8	69.7	61.4	63.8	56.8	68.4	72.2	64.65
F An Illustration for The Proposed Self-prompting Strategy
We offer an illustration (Figure F11) to assist readers in gaining a clearer understanding of the self-prompting strategy.
Figure D9: 3D Segmentation results based on the vanilla NeRF on the LLFF dataset.
Figure E10: Some ground-truth masks (shown in green) and their corresponding instances from the Replica dataset. These unreasonable segmentation targets are filtered out in evaluation.
Table E10: The selected id lists of Replica.
Scenes	ID list
office0	3,4,7,8,9,10,12,14,17,19,21,23,26,28,29,30,36,37,40,42,44,46,54,55,57,58,61
office1	3,7,9,11,13,14,15,17,23,24,29,32,33,36,37,39,42,44,45,46
office2	0,2,8,9,13,14,17,19,23,27,40,41,47,49,51,54,58,60,65,67,70,71,72,73,78,85,90,92,93
office3	3,8,11,14,15,18,19,25,29,30,32,33,38,39,43,51,54,55,61,65,72,76,78,82,87,91,95,96,101,111
office4	1,2,6,7,9,11,17,22,23,26,33,34,39,47,49,51,52,53,55,56
room0	5,6,7,10,13,14,16,25,32,33,35,46,51,53,55,60,64,67,68,83,86,87,92
room1	1,2,4,6,7,9,10,11,16,18,24,28,32,36,37,44,48,52,54,56
room2	3,5,6,7,8,9,11,12,16,18,22,26,27,37,38,39,40,43,49,55,56
In the self-prompting strategy, prompt points  $\mathcal{P}_s$  are derived from an incomplete 2D rendered mask  $\mathbf{M}^{(n)}$ , which is represented as a confidence score map. Initially, the selected prompt points set  $\mathcal{P}_s$  is empty, and the first prompt point  $\mathbf{p}_0$  is selected as the one with the highest confidence score in the mask  $\mathbf{M}^{(n)}$ . For subsequent points, square regions centered around existing prompt points are masked out on  $\mathbf{M}^{(n)}$ . The depth  $z(\mathbf{p})$ , estimated by the pre-trained NeRF, helps convert 2D pixel  $\mathbf{p}$  into a 3D point  $\mathcal{G}(\mathbf{p})$ . The new prompt point is expected to have a high confidence score while being close to existing prompt points. Hence, a distance-aware decay term is introduced to compute the confidence score. The remaining point with the highest decayed score is added to the prompt set. This selection process is repeated until either the number of prompts  $|\mathcal{P}_s|$  reaches a predefined threshold  $n_p$  or the maximum value of the remaining points is less than 0. Please refer to Section 3.4 of the main manuscript for more details.
G The Effect of Occlusion in NeRF
There are two cases of occlusion in NeRFs: part of the target object does not appear in some views (but appear in other views), or it does not appear at all. For the former case, SA3D can recover it using information from other views; for the latter case, there can be parts missing in the target object (see Figure G12). This is an interesting future direction (e.g., applying generative models such as diffusion models).
H More Visualization Results
We present additional visualization results in Figure H13 and Figure H15, showcasing the effectiveness of SA3D across various input prompts. We also provide some visualization of extracted meshes of the segmented objects (Figure H14) to show the extracted 3D geometry. Please note that the quality of these meshes can be further improved by applying more effective NeRF2Mesh methods [56, 66].
Figure F11: An illustration of the self-prompting strategy.
Figure G12: The effect of different kinds of occlusion in NeRF.
Figure H13: Text prompt based visualization results on the LERF figurines dataset.
Figure H14: Some visualization results of the extracted meshes of the segmented objects.
Figure H15: More visualization results on the LLFF dataset and the 360 dataset (based on point and box prompts).