SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiazhong Cen $^{1\dagger}$ , Zanwei Zhou $^{1\dagger}$ , Jiemin Fang $^{2,3\dagger}$ , Chen Yang $^{1}$ , Wei Shen $^{1\boxtimes}$ , Lingxi Xie $^{2}$ , Dongsheng Jiang $^{2}$ , Xiaopeng Zhang $^{2}$ , Qi Tian $^{2}$ $^{1}$  MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University  
 $^{2}$  Huawei Inc.  $^{3}$  School of EIC, Huazhong University of Science and Technology  
wei.shen@sjtu.edu.cn
Recently, the Segment Anything Model (SAM) emerged as a powerful vision foundation model which is capable to segment anything in 2D images. This paper aims to generalize SAM to segment 3D objects. Rather than replicating the data acquisition and annotation procedure which is costly in 3D, we design an efficient solution, leveraging the Neural Radiance Field (NeRF) as a cheap and off-the-shelf prior that connects multi-view 2D images to the 3D space. We refer to the proposed solution as SA3D, for Segment Anything in 3D. It is only required to provide a manual segmentation prompt (e.g., rough points) for the target object in a single view, which is used to generate its 2D mask in this view with SAM. Next, SA3D alternately performs mask inverse rendering and cross-view self-prompting across various views to iteratively complete the 3D mask of the target object constructed with voxel grids. The former projects the 2D mask obtained by SAM in the current view onto 3D mask with guidance of the density distribution learned by the NeRF; The latter extracts reliable prompts automatically as the input to SAM from the NeRF-rendered 2D mask in another view. We show in experiments that SA3D adapts to various scenes and achieves 3D segmentation within minutes. Our research reveals a potential methodology to lift the ability of a 2D vision foundation model to 3D, as long as the 2D model can steadily address promptable segmentation across multiple views. Our code is available at https://github.com/Jumpat/SegmentAnythingin3D.
The computer vision community has been pursuing a vision foundation model that can perform basic tasks (e.g., segmentation) in any scenario and for either 2D or 3D image data. Recently, the Segment Anything Model (SAM) [25] emerged and attracted a lot of attention, due to its ability to segment anything in 2D images, but generalizing the ability of SAM to 3D scenes remains mostly uncovered. One may choose to replicate the pipeline of SAM to collect and semi-automatically annotate a large set of 3D scenes, but the costly burden seems unaffordable for most research groups.
We realize that an alternative and efficient solution lies in equipping the 2D foundation model (i.e., SAM) with 3D perception via a 3D representation model. In other words, there is no need to establish a 3D foundation model from scratch. However, there is a prerequisite: the 3D representation model shall be capable to render 2D views and register 2D segmentation results to the 3D scene. Thus, we use the Neural Radiance Fields (NeRF) [38, 53, 3] as an off-the-shelf solution. NeRF is a family of algorithms that formulates each 3D scene into a deep neural network that serves as a 3D prior connecting multiple 2D views.
Figure 1: Given any pre-trained NeRF, SA3D takes prompts from one single rendered view as input and outputs the 3D segmentation result for the specific target.
As shown in Figure 1, our solution is named Segment Anything in 3D (SA3D). Given a NeRF trained on a set of 2D images, SA3D takes prompts (e.g., click points on the object) in a single rendered view as input, which is used to generate a 2D mask in this view with SAM. Next, SA3D alternately performs two steps across various views to iteratively complete the 3D mask of the object constructed with voxel grids. In each round, the first step is mask inverse rendering, in which the previous 2D segmentation mask obtained by SAM is projected onto the 3D mask via density-guided inverse rendering offered by the NeRF. The second step is cross-view self-prompting, in which NeRF is used to render the 2D segmentation mask (which may be inaccurate) based on the 3D mask and the image from another view, then a few point prompts are automatically generated from the rendered mask and fed into SAM to produce a more complete and accurate 2D mask. The above procedure is executed iteratively until all necessary views have been sampled.
We conduct various (e.g., object, part-level) segmentation tasks on the Replica [51] and NVOS [47] datasets. Without re-training/re-designing SAM or NeRF, SA3D easily and efficiently adapts to different scenarios. Compared to existing approaches, SA3D enjoys a simplified pipeline that typically completes 3D segmentation within minutes. SA3D not only offers an efficient tool for segmenting anything in 3D, but also reveals a generic methodology to lift 2D foundation models to the 3D space. The only prerequisite lies in the ability to steadily address promptable segmentation across multiple views, and we hope it becomes a general property of 2D foundation models in the future.
2D Segmentation Since FCN [36] has been proposed, research on 2D image segmentation has experienced a rapid growth. Various sub-fields of segmentation have been explored deeply by numerous studies [18, 24, 4, 71]. With transformers [58, 10] entering the field of segmentation, many new segmentation architectures [72, 7, 6, 52, 63] have been proposed and the whole field of segmentation has been further developed. A recent significant breakthrough in this field is the Segment Anything Model (SAM) [25]. As an emerging vision foundation model, SAM is recognized as a potential game-changer, which aims to unify the 2D segmentation task by introducing a prompt-based segmentation paradigm. An analogous model to SAM is SEEM [75], which also exhibits impressive open-vocabulary segmentation capabilities.
3D Segmentation Numerous methods have explored various types of 3D representations to perform 3D segmentation. These scene representations include RGB-D images [60, 62, 64, 8], point clouds [44, 45, 70] and grid space such as voxels [21, 55, 35], cylinders [74] and bird's eye view space [67, 16]. Although 3D segmentation has been developed for a period of time, compared with 2D segmentation, the scarcity of labeled data and high computational complexity make it difficult to design a unified framework similar to SAM.
Lifting 2D Vision Foundation Models to 3D To tackle the limitation of data scarcity, many previous studies [23, 43, 9, 17, 69, 31, 65, 22] explored lifting 2D foundation models to 3D. In these studies, the most relevant work to SA3D is LERF [23], which trains a feature field of the Vision-Language Model (i.e., CLIP [46]) together with the radiance field. Compared with SA3D,
LERF focuses on coarsely localizing the specific objects with text prompts but not fine-grained 3D segmentation. The reliance on CLIP features makes it insensitive to the specific location information of the target object. When there are multiple objects with similar semantics in the scene, LERF cannot perform effective 3D segmentation. The remaining methods mainly focus on point clouds. By connecting the 3D point cloud with specific camera poses with 2D multi-view images, the extracted features by 2D foundation models can be projected to the 3D point cloud. The data acquisition of these methods is more expensive than ours, i.e., acquiring multi-view images for NeRFs.
Segmentation in NeRFs Neural Radiance Fields (NeRFs) [38, 53, 3, 1, 40, 19, 13, 61, 30, 12] are a series of 3D implicit representation. Inspired by their success in 3D consistent novel view synthesis, numerous studies have delved into the realm of 3D segmentation within NeRFs. Zhi et al. [73] proposes Semantic-NeRF, a method that incorporates semantics into appearance and geometry. They showcase the potential of NeRFs in label propagation and refinement. NVOS [47] introduces an interactive approach to select 3D objects from NeRFs by training a lightweight multi-layer perception (MLP) using custom-designed 3D features. Other approaches, e.g. N3F [57], DFF [27], LERF [23] and ISRF [15], aim to lift 2D visual features to 3D through training additional feature fields. These methods are required to re-design-/train NeRF models and usually involve additional feature-matching processes. There are also some other instance segmentation and semantic segmentation approaches [50, 41, 11, 68, 34, 20, 2, 14, 59, 28] combined with NeRFs.
The most closely related approach to our SA3D is MVSeg [39], a component of SPIn-NeRF [39], which focuses on NeRF inpainting. MVSeg adopts video segmentation techniques to propagate a 2D mask across different views and employs these masks as labels for training a Semantic-NeRF model. However, video segmentation models lack explicit 3D structure information, which are hard to handle significant occlusions in complex scenes. Our method aims at building NeRF-driven consistency across views based on self-prompting and lifting 2D masks to robust 3D masks.
In this section, we first give a brief review of Neural Radiance Fields (NeRFs) and the Segment Anything Model (SAM). Then we introduce the overall pipeline of SA3D. Finally, we demonstrate the design of each component in SA3D in detail.
Neural Radiance Fields (NeRFs) Given a training dataset  $\mathcal{L}$  of multi-view 2D images, NeRFs [38] learn a function  $f_{\theta}:(\mathbf{x},\mathbf{d})\to (\mathbf{c},\sigma)$ , which maps the spatial coordinates  $\mathbf{x}\in \mathbb{R}^3$  and the view direction  $\mathbf{d}\in \mathbb{S}^2$  of a point into the corresponding color  $\mathbf{c}\in \mathbb{R}^3$  and volume density  $\sigma \in \mathbb{R}$ .  $\pmb{\theta}$  denotes the learnable parameters of the function  $f$  which is usually represented by a multi-layer perceptron (MLP). To render an image  $\mathbf{I}_{\theta}$ , each pixel undergoes a ray casting process where a ray  $\mathbf{r}(t) = \mathbf{x}_o + t\mathbf{d}$  is projected through the camera pose. Here,  $\mathbf{x}_o$  is the camera origin,  $\mathbf{d}$  is the ray direction, and  $t$  denotes the distance of a point along the ray from the origin. The RGB color  $\mathbf{I}_{\theta}(\mathbf{r})$  at the location determined by ray  $\mathbf{r}$  is obtained via a differentiable volume rendering algorithm:
$$
\mathbf {I} _ {\boldsymbol {\theta}} (\mathbf {r}) = \int_ {t _ {n}} ^ {t _ {f}} \omega (\mathbf {r} (t)) \mathbf {c} (\mathbf {r} (t), \mathbf {d}) \mathrm {d t}, \tag {1}
$$
where  $\omega (\mathbf{r}(t)) = \exp (-\int_{t_n}^t\sigma (\mathbf{r}(s))\mathrm{d}s)\cdot \sigma (\mathbf{r}(t))$  and  $t_n$  and  $t_f$  denote the near and far bounds of the ray, respectively.
Segment Anything Model (SAM) SAM [25] takes an image  $\mathbf{I}$  and a set of prompts  $\mathcal{P}$  as input, and outputs the corresponding 2D segmentation mask  $\mathbf{M}_{\mathrm{SAM}}$  in the form of a bitmap, i.e.,
$$
\mathbf {M} _ {\mathrm {S A M}} = s (\mathbf {I}, \mathcal {P}). \tag {2}
$$
The prompts  $\mathbf{p} \in \mathcal{P}$  can be points, boxes, texts, and masks.
3.2 Overall Pipeline
We assume that we already have a NeRF model trained on the dataset  $\mathcal{I}$ . Throughout this paper, unless otherwise specified, we opt to employ the TensoRF [3] as the NeRF model, considering its
Figure 2: The overall pipeline of SA3D. Given a NeRF trained on a set of multi-view 2D images, SA3D first takes prompts in a single view for the target object as input and uses SAM to produce a 2D mask in this view with these prompts. Then, SA3D performs an alternated process of mask inverse rendering and cross-view self-prompting to complete the 3D mask of the target object constructed with voxel grids. Mask inverse rendering is performed to project the 2D mask obtained by SAM onto the 3D mask according to the learned density distribution embedded in the NeRF. Cross-view self-prompting is conducted to extract reliable prompts automatically as the input to SAM from the NeRF-rendered 2D mask given a novel view. This alternated process is executed iteratively until we get the complete 3D mask.
superior efficiency of training and rendering. As shown in Figure 2, an image  $\mathbf{I}^{\mathrm{in}}$  from a specific view is first rendered with the pre-trained NeRF model. A set of prompts (e.g., in this paper, we often use a set of points),  $\mathcal{P}^{\mathrm{in}}$ , is introduced and fed into SAM along with the rendered image. The 2D segmentation mask  $\mathbf{M}_{\mathrm{SAM}}^{\mathrm{in}}$  of the according view is obtained, which is then projected onto the 3D mask  $\mathbf{V} \in \mathbb{R}^{3}$  constructed voxel grids with the proposed mask inverse rendering technique (Section 3.3). Then a 2D segmentation mask  $\mathbf{M}^{(n)}$  from a novel view is rendered from the 3D mask. The rendered mask is usually inaccurate. We propose a cross-view self-prompting method (Section 3.4) to extract point prompts  $\mathcal{P}^{(n)}$  from the rendered mask and further feed them into SAM. Thus a more accurate 2D mask  $\mathbf{M}_{\mathrm{SAM}}^{(n)}$  in this novel view is produced and also projected onto the voxel grids to complete the 3D mask. The above procedure is executed iteratively with more views traversed. Meanwhile, the 3D mask become more and more complete. The whole process bridges 2D segmentation results with 3D ones efficiently. Noting that no neural network needs to be optimized except the 3D mask.
3.3 Mask Inverse Rendering
As shown in Equation (1), the color of each pixel in a rendered image is determined by a sum of weighted colors along the corresponding ray. The weight  $\omega (\mathbf{r}(t))$  reveals the object structure within the 3D space, where a high weight indicates the corresponding point close to the object's surface. Mask inverse rendering aims to project the 2D mask to the 3D space to form the 3D mask based on these weights.
Formally, the 3D mask is represented as voxel grids  $\mathbf{V} \in \mathbb{R}^{L \times W \times H}$ , where each grid vertex stores a zero-initialized soft mask confidence score. Based on these voxels grids, each pixel of the 2D mask from one view is rendered as
$$
\mathbf {M} (\mathbf {r}) = \int_ {t _ {n}} ^ {t _ {f}} \omega (\mathbf {r} (t)) \mathbf {V} (\mathbf {r} (t)) \mathrm {d t}, \tag {3}
$$
where  $\mathbf{r}(t)$  is the ray casting through the mask pixel,  $\omega (\mathbf{r}(t))$  is inherited from density values of the pre-trained NeRF, and  $\mathbf{V}(\mathbf{r}(t))$  denotes the mask confidence score at the location  $\mathbf{r}(t)$  obtained from voxel grids  $\mathbf{V}^1$ . Denote  $\mathbf{M}_{\mathrm{SAM}}(\mathbf{r})$  as the corresponding mask generated by SAM. When  $\mathbf{M}_{\mathrm{SAM}}(\mathbf{r}) = 1$
${}^{1}\mathbf{V}(\mathbf{r}(t))$  is computed by trilinearly interpolating vertex values of the 3D mask grids.
the goal of mask inverse rendering is to increase  $\mathbf{V}(\mathbf{r}(t))$  with respect to  $\omega (\mathbf{r}(t))$ . In practice, this can be optimized using the gradient descent algorithm. For this purpose, we define the mask projection loss as the negative product between  $\mathbf{M}_{\mathrm{SAM}}(\mathbf{r})$  and  $\mathbf{M}(\mathbf{r})$ :
$$
\mathcal {L} _ {\mathrm {p r o j}} = - \sum_ {\mathbf {r} \in \mathcal {R} (\mathbf {I})} \mathbf {M} _ {\mathrm {S A M}} (\mathbf {r}) \cdot \mathbf {M} (\mathbf {r}), \tag {4}
$$
where  $\mathcal{R}(\mathbf{I})$  denotes the ray set of the image  $\mathbf{I}$ .
The mask projection loss is constructed based on the assumption that both the geometry from the NeRF and the segmentation results of SAM are accurate. However, in practice, this is not always the case. We append a negative refinement term to the loss to optimize the 3D mask grids according to multi-view mask consistency:
$$
\mathcal {L} _ {\mathrm {p r o j}} = - \sum_ {\mathbf {r} \in \mathcal {R} (\mathbf {I})} \mathbf {M} _ {\mathrm {S A M}} (\mathbf {r}) \cdot \mathbf {M} (\mathbf {r}) + \lambda \sum_ {\mathbf {r} \in \mathcal {R} (\mathbf {I})} (1 - \mathbf {M} _ {\mathrm {S A M}} (\mathbf {r})) \cdot \mathbf {M} (\mathbf {r}), \tag {5}
$$
where  $\lambda$  is a hyper-parameter to determine the magnitude of the negative term. With this negative refinement term, only if SAM consistently predicts a region as foreground from different views, SA3D marks its corresponding 3D region as foreground. In each iteration, the 3D mask  $\mathbf{V}$  is updated via  $\mathbf{V} \gets \mathbf{V} - \eta \frac{\partial \mathcal{L}_{\mathrm{proj}}}{\partial \mathbf{V}}$  with gradient descent, where  $\eta$  denotes the learning rate.
3.4 Cross-view Self-prompting
Mask inverse rendering enables projecting 2D masks into the 3D space to form the 3D mask of a target object. To construct accurate 3D mask, substantial 2D masks from various views need to be projected. SAM can provide high-quality segmentation results given proper prompts. However, manually selecting prompts from every view is time-consuming and impractical. We propose a cross-view self-prompting mechanism to produce prompts for different novel views automatically.
Specifically, we first render a novel-view 2D segmentation mask  $\mathbf{M}^{(n)}$  from the 3D mask grids  $\mathbf{V}$  according to Equation (3). This mask is usually inaccurate, especially at the preliminary iteration of SA3D. Then we obtain some point prompts from the rendered mask with a specific strategy. The above process is named cross-view self-prompting. While there are multiple possible solutions for this strategy, we present a feasible one that has been demonstrated to be effective.
Self-prompting Strategy Given an inaccurate 2D rendered mask  $\mathbf{M}^{(n)}$ , the self-prompting strategy aims to extract a set of prompt points  $\mathcal{P}_s$  from it, which can help SAM to generate 2D segmentation result as accurate as possible. It is important to note that  $\mathbf{M}^{(n)}$  is not a typical 2D bitmap, but rather a confidence score map computed using Equation (3). Since each image pixel  $\mathbf{p}$  corresponds to a ray  $\mathbf{r}$  in a rendered view, we use  $\mathbf{p}$  for an easier demonstration of the prompt selection strategy on images.
As  $\mathcal{P}_s$  is initialized to an empty set, the first prompt point  $\mathbf{p}_0$  is selected as the point with the highest mask confidence score:  $\mathbf{p}_0 = \arg \max_{\mathbf{p}} \mathbf{M}^{(n)}(\mathbf{p})$ . To select new prompt points, we first mask out square shaped regions² on  $\mathbf{M}^{(n)}$  centered with each existing point prompt  $\hat{\mathbf{p}} \in \mathcal{P}_s$ . Considering the depth  $z(\mathbf{p})$  can be estimated by the pre-trained NeRF, we transform the 2D pixel  $\mathbf{p}$  to the 3D point  $\mathcal{G}(\mathbf{p}) = (x(\mathcal{G}(\mathbf{p})), y(\mathcal{G}(\mathbf{p})), z(\mathcal{G}(\mathbf{p})))$ :
$$
\binom {x (\mathcal {G} (\mathbf {p}))} {y (\mathcal {G} (\mathbf {p}))} = z (\mathbf {p}) \mathbf {K} ^ {- 1} \binom {x (\mathbf {p})} {y (\mathbf {p})} \tag {6}
$$
where  $x(\mathbf{p}), y(\mathbf{p})$  denote the 2D coordinates of  $\mathbf{p}$ , and  $\mathbf{K}$  denotes the camera intrinsic. The new prompt point is expected to have a high confidence score while being close to existing prompt points. Considering the two factors, we introduce a decay term to the confidence score. Let  $d(\cdot, \cdot)$  denote the min-max normalized Euclidean distance. For each remaining point  $\mathbf{p}$  in  $\mathbf{M}^{(n)}$ , the decay term is
$$
\Delta \mathbf {M} ^ {(n)} (\mathbf {p}) = \min  \{\mathbf {M} ^ {(n)} (\hat {\mathbf {p}}) \cdot d (\mathcal {G} (\mathbf {p}), \mathcal {G} (\hat {\mathbf {p}})) \mid \hat {\mathbf {p}} \in \mathcal {P} _ {s} \}. \tag {7}
$$
Then a decayed mask confidence score  $\tilde{\mathbf{M}}^{(n)}(\mathbf{p})$  is computed as
$$
\tilde {\mathbf {M}} ^ {(n)} (\mathbf {p}) = \mathbf {M} ^ {(n)} (\mathbf {p}) - \Delta \mathbf {M} ^ {(n)} (\mathbf {p}). \tag {8}
$$
2The side length of the region is set as the radius of a circle whose area is equal to the rendered mask  $\mathbf{M}^{(n)}$
Figure 3: Some visualization results in different scenes (LERF-donuts [23], LERF-figurines, Replica-room0 [51] and 360-kitchen [1]).
The remaining point with the highest decayed score, i.e.,  $\mathbf{p}^{*} = \arg \max_{\mathbf{p}\notin \mathcal{P}_{s}}\tilde{\mathbf{M}}^{(n)}(\mathbf{p})$  , is added to the prompt set:  $\mathcal{P}_s = \mathcal{P}_s\cup \{\mathbf{p}^*\}$  . The above selection process is repeated until either the number of prompts  $|\mathcal{P}_s|$  reaches a predefined threshold  $n_p$  or the maximum value of  $\tilde{\mathbf{M}}^{(n)}(\mathbf{p})$  is smaller than 0.
IoU-aware View Rejection When the target object is rendered in heavily occluded views, SAM may produce incorrect segmentation results and degrades the quality of the 3D mask. To avoid such situations, we introduce an additional view rejection mechanism based on the intersection-over-union (IoU) between the rendered mask  $\mathbf{M}^{(n)}$  and the SAM prediction  $\mathbf{M}_{\mathrm{SAM}}^{(n)}$ . If the IoU falls below a predefined threshold  $\tau$ , it indicates a poor overlap between the two masks. The prediction from SAM is rejected, and the mask inverse rendering step is skipped in this iteration.
In this section, we quantitatively evaluate the segmentation ability of SA3D on various datasets. Then, we qualitatively demonstrate the versatility of SA3D, which can conduct instance segmentation, part segmentation, and text-prompted segmentation etc.
For quantitative experiments, we use the Neural Volumetric Object Selection (NVOS) [47], SPIn-NeRF [39], and Replica [51] datasets. The NVOS [47] dataset is based on the LLFF dataset [37], which includes several forward-facing scenes. For each scene, NVOS provides a reference view
with scribbles and a target view with 2D segmentation masks annotated. Similar to NVOS, SPIn-NeRF [39] annotates some data manually to evaluate interactive 3D segmentation performance. These annotations are based on some widely-used NeRF datasets [37, 38, 29, 26, 13]. The Replica [51] dataset provides high-quality reconstruction ground truths of various indoor scenes, including clean dense geometry, high-resolution and high-dynamic-range textures, glass and mirror surface information, semantic classes, planar segmentation, and instance segmentation masks. For qualitative analysis, we use the LLFF [37] dataset and the  $360^{\circ}$  dataset [1]. SA3D is further applied to the LERF [23] dataset, which contains more realistic and challenging scenes.
NVOS Dataset For fair comparisons, we follow the experimental setting of the original NVOS [47]. We first scribble on the reference view (provided by the NVOS dataset) to conduct 3D segmentation, and then render the 3D segmentation result on the target view and evaluate the IoU and pixel-wise accuracy with the provided ground truth. Note that the scribbles are preprocessed to meet the requirements of SAM. More details can be found in the supplement.
Table 1: Quantitative results on NVOS.
Method	mIoU (%)	mAcc (%)
Graph-cut (3D) [48, 47]	39.4	73.6
NVOS [47]	70.1	92.0
ISRF [15]	83.8	96.4
SA3D (ours)	90.3	98.2
As shown in Table 1, SA3D outperforms previous approaches by large margins, i.e.,  $+6.5\mathrm{mIoU}$  over the previous SOTA ISRF and  $+20.2\mathrm{mIoU}$  over the NVOS.
Table 2: Quantitative results on the SPIn-NeRF dataset.
Scenes	Single view	MVSeg [39]	SA3D (ours)
IoU (%)	Acc (%)	IoU (%)	Acc (%)	IoU (%)	Acc (%)
Orchids	79.4	96.0	92.7	98.8	83.6	96.9
Leaves	78.7	98.6	94.9	99.7	97.2	99.9
Fern	95.2	99.3	94.3	99.2	97.1	99.6
Room	73.4	96.5	95.6	99.4	88.2	98.3
Horns	85.3	97.1	92.8	98.7	94.5	99.0
Fortress	94.1	99.1	97.7	99.7	98.3	99.8
Fork	69.4	98.5	87.9	99.5	89.4	99.6
Pinecone	57.0	92.5	93.4	99.2	92.9	99.1
Truck	37.9	77.9	85.2	95.1	90.8	96.7
Lego	76.0	99.1	74.9	99.2	92.2	99.8
mean	74.6	95.5	90.9	98.9	92.4	98.9