SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanlin Chen Chen Li Mengqi Guo Zhiwen Yan Gim Hee Lee
Department of Computer Science, National University of Singapore
{hanlin.chen, gimhee.lee}@comp.nus.edu.sg
3D scene segmentation based on neural implicit representation has emerged recently with the advantage of training only on 2D supervision. However, existing approaches still requires expensive per-scene optimization that prohibits generalization to novel scenes during inference. To circumvent this problem, we introduce a generalizable 3D segmentation framework based on implicit representation. Specifically, our framework takes in multi-view image features and semantic maps as the inputs instead of only spatial information to avoid overfitting to scene-specific geometric and semantic information. We propose a novel soft voting mechanism to aggregate the 2D semantic information from different views for each 3D point. In addition to the image features, view difference information is also encoded in our framework to predict the voting scores. Intuitively, this allows the semantic information from nearby views to contribute more compared to distant ones. Furthermore, a visibility module is also designed to detect and filter out detrimental information from occluded views. Due to the generalizability of our proposed method, we can synthesize semantic maps or conduct 3D semantic segmentation for novel scenes with solely 2D semantic supervision. Experimental results show that our approach achieves comparable performance with scene-specific approaches. More importantly, our approach can even outperform existing strong supervision-based approaches with only 2D annotations. Our source code is available at: https://github.com/HLinChen/GNeSF.
Understanding high-level semantics of 3D scenes in digital images and videos is a fundamental research in computer vision. Several extensively studied tasks such as scene classification [22], object detection [34], and semantic segmentation [10] facilitate the extraction of semantic descriptions from RGB and other sensor data. These tasks form the foundation for applications such as visual navigation [4] and robotic interaction [2].
Several commonly used 3D representations include 3D point clouds [11, 32], voxel grids [42], and polygonal meshes [28, 18]. A direct learning of semantics on these 3D representations require large amounts of well-annotated 3D ground truths for training. Unfortunately, these 3D ground truths are often much more laborious and expensive to obtain compared to its 2D counterparts. Recently, Neural Radiance Field (NeRF) [30, 52] has emerged as a new 3D representation, which can capture intricate geometric details [41] using solely RGB images. Many researchers utilize this 3D representation to propose the concept of Neural Semantic Field [55, 17, 3], where semantics is assigned to each 3D point. However, these existing works on neural semantic fields inherited the per-scene optimization limitation of the vanilla NeRF and thus cannot generalize across novel scenes. As a result, the practicality of the approaches for semantic segmentation is severely restricted.
In view of the limitations in existing neural semantic field works, this paper addresses the challenging task of generalizable neural semantic fields. The objective is to construct a generalizable semantic field capable of producing high-quality semantic rendering from novel views and 3D semantic
segmentation using input images from unseen scenes. The generalization ability offers numerous interactive applications that provide valuable feedback to users during 3D capture without the requirement for retraining. Recently, NeSF [44] claims the ability to generate 3D semantic fields for novel scenes by utilizing 3D UNet [36] to extract semantic features from a density field obtained from a pre-trained NeRF model. However, this method lacks efficiency as it requires training a separate NeRF model for each scene before segmenting the 3D scene. Furthermore, it lacks sufficient generalizability to new scenes, thus making it suitable only for offline applications. Recently, several researchers propose generalizable NeRF methods [6, 45, 53] that demonstrate effective generalization across scenes for reconstructing radiance fields without the need for laborious per-scene optimization. A naive approach for generalizable Neural Semantic Field (NeSF) is to incorporate an additional semantic head to the generalizable NeRF framework. However, this method does not perform well for complex and realistic indoor scenes, as demonstrated in Tab. 1 and [20]. One possible reason is that semantics represents high-level information, and a simple head without direct 3D semantic supervision is insufficient for generating 3D semantic fields.
Inspired by image-based rendering (IBR) [9, 45], we propose a soft voting scheme (shown in Fig. 2a) to aggregate the 2D semantic information from different views for each 3D point for learning a Generalizable Neural Semantic Field (GNeSF). Specifically, we use a pre-trained 2D semantic segmentation model to generate 2D semantic maps for the source views and then leverage the warping, resampling, and/or blending operations of IBR to infer the semantics of a sample 3D point. Considering the unbounded nature and the lack of meaning for logit values across multiple views, we utilize a probability distribution instead of semantic logits to represent semantics across views. Our framework utilizes multi-view image features instead of position information as input to further enhance the generalizability of our model and prevent overfitting to scene-specific geometric and semantic information. In addition to the extracted image features, our framework incorporates view difference information to include the prior of higher significance for nearby views when generating the voting scores for blending source views. We then introduce a soft voting scheme to compute the semantics of each sample point as the weighted combination of the projected semantics from the source views.
Geometric information is predicted to perform volumetric rendering after obtaining semantics of 3D points along sampled ray. Specifically, we first build a feature volume by projecting each vertex of the volume to latent 2D features and aggregating these 2D features to get a density 3D feature. This feature volume is then fed into a 3D neural network to obtain a geometry encoding volume. For a sampled 3D point, we use an MLP network to infer the specific geometric representations, e.g., density or Signed Distance Function (SDF), from the interpolated features of the geometry encoding volume.
Weights of points along the ray are obtained from the predicted geometric representations. A final semantic value is computed for each ray through volume rendering by weighted summing semantics alone the ray. Due to the generalizability of our proposed method, we can synthesize semantic maps or conduct 3D semantic segmentation for novel scenes with solely 2D semantic supervision. To facilitate 3D semantic segmentation, we design a visibility module (shown in Fig. 2b) that identifies and removes occluded views for each spatial point. The remaining semantics from the visible views are then utilized to vote on the semantics of the respective point. In summary, our contributions include:
- We are the first to tackle the task of generalizable neural semantic field. Our proposed GNeSF is capable of inferring novel view synthesis of semantic maps or 3D semantic segmentation of novel scenes without retraining. Furthermore, our method depends solely on 2D semantic supervision during training.
- We propose a novel soft voting scheme that determines the semantics for each 3D point by considering the 2D semantics from multiple source views. A visibility module is also designed to detect and filter out detrimental information from occluded views.
- Our proposed GNeSF enables synthesis of semantic maps for both novel views and novel scenes, and achieves comparable performance to per-scene optimized methods. The results on ScanNet and Replica demonstrate comparable or superior performance compared to existing methods utilizing strong 3D supervision.
2.1 2D & 3D Semantic Segmentation
Traditionally, semantic segmentation has been treated as a per-pixel classification task, where FCN-based architectures [27] assign category labels to individual pixels independently. Subsequent methods have acknowledged the importance of context in achieving accurate per-pixel classification and have concentrated on developing specialized context modules [7, 8, 54] or incorporating self-attention variations [39, 46, 10]. In our work, we utilize the state-of-the-art Mask2Former [10] with a Swin-Transformer [26] backbone as our pre-trained 2D semantic segmenter. 3D semantic segmentation also has been investigated with pre-computed 3D structures such as voxel grids or point clouds [19, 32, 33, 37, 12, 14, 35], and simultaneous segmentation and 3D reconstruction from 2D images [31, 29]. Unlike these methods, our GNeSF aims to segment a dense 3D representation using only 2D inputs and semantic supervision without the need for 3D semantic annotations.
2.2 Neural Semantic Fields
Semantic-NeRF [55] is first to introduce the integration of semantics into NeRF. It showcases the fusion of noisy 2D semantic segmentations into a coherent volumetric model. This integration enhanced the accuracy of the model and enabled the synthesis of novel views with semantic masks. Subsequently, numerous studies have built upon this concept. For example, [17, 23, 21] incorporate instance modeling, and [43] encode abstract visual features that allow for post hoc derivation of semantic segmentation. NeRF-SOS [16] combines object segmentation and neural radiance fields to segment objects in complex real-world scenes using self-supervised learning. [50] proposes an object-compositional neural radiance field. RFP [25] designs an unsupervised approach to segment objects in 3D during reconstruction using only unlabeled multi-view images of a scene. Panoptic NeRF [17] and DM-NeRF [3] are specifically designed for panoptic radiance fields for tasks such as label transfer and scene editing, respectively. However, these methods require scene-specific optimization and thus not generalizable to unseen scenes. Similar to our objective, NeSF [44] addresses the challenge of generalization to unseen scenes. Nonetheless, NeSF relies on multiple pre-trained and scene-specific optimized NeRF models for density field generation and segmentation which restricts its overall generalizability. In contrast, our GNeSF is able to infer on novel scenes directly without any further optimization.
2.3 Generalizable Neural Implicit Representation
Significant advancements have been achieved in the field of neural implicit representation [30, 48, 52]. However, these approaches heavily depend on computationally expensive per-scene optimization. To address this limitation, several new neural implicit representation methods emphasizing on generalization has emerged. These methods [6, 45, 49, 51, 5] aim to learn a representation of the radiance field from a provided set of images of novel scenes. This enables the synthesis of novel views without requiring per-scene optimization. PixelNeRF [51] and IBRNet [45] utilize volume rendering techniques to generate novel view images by employing warped features from nearby reference images. In addition to novel synthesis, MonoNeuralFusion [56] introduces a novel neural implicit scene representation with volume rendering for high-fidelity generalizable 3D scene reconstruction from monocular videos. Inspired by these generalizable methods, we takes in multi-view image feature as the input instead of the position information to enhance the generalizability of our model. Semantic knowledge obtained from 2D vision is then aggregated to infer 3D semantics by the proposed soft voting scheme.
3 Our GNeSF: Generalizable Neural Semantic Fields
We propose GNeSF to achieve generalizable semantic segmentation on neural implicit representation. As illustrated in Fig. 1, our GNeSF consists of three main components: 1) Feature extraction and semantic prediction from multiple source views; 2) Geometry and semantics prediction for sampled 3D points; 3) Semantic map prediction with volume rendering.
Figure 1: The GNeSF framework comprises three key components: 1) Extraction of features and semantic prediction from multiple source views, shown in the blue dashed box; 2) Prediction of geometry and semantics for sampled 3D points, shown in the yellow dashed box; 3) Prediction of semantic maps using volume rendering, shown in the red dashed box. Here, "MVS" means using Multi-View Stereo to build the feature volume.
3.1 Source-View Image Features and Semantics Map
Inspired by existing generalizable neural implicit representations [6, 45], our framework leverages image features from source-view images for generalization to unseen scenes during inference. We extract an image feature map  $\mathbf{F}_i\in \mathbb{R}^{H_i\times W_i\times d}$  for each source image  $\mathbf{I}_i\in [0,1]^{\bar{H}_i\times W_i\times 3}$  with a U-Net-based convolutional neural network, where  $H_{i}\times W_{i}$  and  $d$  denotes the dimensions of the image and the feature maps, respectively. Concurrently, we use a pre-trained mask2former [10] to predict the semantic maps  $\mathbf{S}_i\in \mathbb{R}^{H_i\times W_i\times c}$  of each source image  $\mathbf{I}_i$ , where  $c$  represents the number of categories.
3.2 3D Point Geometry and Semantics Predictions
Using the source-view image features and semantics maps, we predict the geometry  $\sigma(\mathbf{x})$  and semantics  $\mathbf{s}(\mathbf{x})$  for each sampled 3D point  $\mathbf{x} \in \mathbb{R}^3$ . We first build a geometry encoding volume  $\mathbf{M}$  based on the image feature map  $\mathbf{F}_i$  and then predict the geometry  $\sigma(\mathbf{x})$  using an MLP. Concurrently, we design a soft voting scheme (shown in Fig. 2a) based on the 2D semantic observations  $\mathbf{S}_i \in \mathbb{R}^{H_i \times W_i \times c}$  to predict the semantics  $\mathbf{s}(\mathbf{x})$ .
3.2.1 Geometry Encoding Volume
To construct the geometry encoding volume  $\mathbf{M}$ , we project each vertex  $\mathbf{k} \in \mathbb{R}^3$  of the volume to the image feature maps  $\mathbf{F}_i$  and obtain its image features  $\mathbf{F}_i(\pi_i(\mathbf{k}))$  by interpolation, where  $\pi_i(\mathbf{k})$  denotes the projected pixel location of  $\mathbf{k}$  on the feature map  $\mathbf{F}_i$ . For simplicity, we abbreviate  $\mathbf{F}_i(\pi_i(\mathbf{k}))$  as  $\mathbf{F}_i(\mathbf{k})$ . We then compute the mean  $\mu \in \mathbb{R}^d$  and variance  $\mathbf{v} \in \mathbb{R}^d$  of the projected image features  $\{\mathbf{F}_i(\mathbf{k})\}_{i=1}^N$  from the  $N$  source views to capture the global information. The mean  $\mu$  and variance  $\mathbf{v}$  are concatenated to build a feature volume  $\mathbf{B}$ . Intuitively, the mean image feature provides cues for the appearance information, and the variance for geometry reasoning. Finally, we use a 3D neural network  $H_g$  to aggregate the feature volume  $\mathbf{B}$  to obtain the geometry encoding volume  $\mathbf{M}$ . Formally, the whole process can be expressed by:
$$
\mu (\mathbf {k}) = \operatorname {M e a n} \left(\left\{\mathbf {F} _ {i} (\mathbf {k}) \right\} _ {i = 1} ^ {N}\right), \quad \mathbf {v} (\mathbf {k}) = \operatorname {V a r} \left(\left\{\mathbf {F} _ {i} (\mathbf {k}) \right\} _ {i = 1} ^ {N}\right), \tag {1a}
$$
$$
\mathbf {B} (\mathbf {k}) = \left[ \mu (\mathbf {k}), \mathbf {v} (\mathbf {k}) \right], \quad \mathbf {M} = H _ {g} (\mathbf {B}), \tag {1b}
$$
where  $[\cdot, \cdot]$  represents feature concatenation, and  $\mathrm{Mean}(\cdot)$  and  $\mathrm{Var}(\cdot)$  are the averaging and variance operations. For a sampled 3D point  $\mathbf{x}$ , we use an MLP network  $f_{\theta}$  to predict the geometry  $\sigma(\mathbf{x})$  based on the interpolated features of the geometry encoding volume  $\mathbf{M}(\mathbf{x})$ , i.e.:
$$
\sigma (\mathbf {x}) = f _ {\theta} (\mathbf {M} (\mathbf {x})). \tag {2}
$$
(a) Soft Voting Scheme
(b) Visibility Module
Figure 2: (a) The soft voting scheme aggregates the 2D semantic information from different views to infer semantics of each 3D point. (b) The visibility module detects and eliminates occluded views for each spatial point. For example,  $\mathbf{x}_1$  is visible in the 2D semantic observation because the SDF values along the ray from the source view to the target point transits from positive to negative. On the other hand,  $\mathbf{x}_2$  and  $\mathbf{x}_3$  are occluded in the 2D semantic observation because the value changes from positive to negative and then back to positive.
Note that geometry  $\sigma (\mathbf{x})$ , e.g., volume density or Signed Distance Function (SDF) is represented differently in different neural implicit representations. More details are provided in Sec. 4.
3.2.2 Semantics Soft Voting
Given a query point location  $\mathbf{x} \in \mathbb{R}^3$  along a ray  $\mathbf{r} \in \mathbb{R}^3$ , we project  $\mathbf{x}$  onto  $N$  nearby source views using the respective camera parameters. Image features  $\{\mathbf{F}_i(\mathbf{x})\}_{i=1}^N \in \mathbb{R}^d$  and semantics maps  $\{\mathbf{S}_i(\mathbf{x})\}_{i=1}^N \in [0,1]^k$  are then sampled at the projected pixel locations over the  $N$  source views through bilinear interpolation on the image feature  $\mathbf{F}_i$  and semantic maps  $\mathbf{S}_i$ , respectively. We propose a soft voting scheme to predict the semantics  $\mathbf{s}(\mathbf{x})$  of a sampled 3D point  $\mathbf{x}$  based on the 2D projected image features  $\{\mathbf{F}_i(\mathbf{x})\}_{i=1}^N$  and semantics maps  $\{\mathbf{S}_i(\mathbf{x})\}_{i=1}^N$ . Specifically, the 2D projected image features  $\{\mathbf{F}_i(\mathbf{x})\}_{i=1}^N$  are used to predict voting weights  $\{v_i\}_{i=1}^N$  for the corresponding semantics maps  $\{\mathbf{S}_i(\mathbf{x})\}_{i=1}^N$  in the source views.
We first calculate the mean and variance of the image features  $\{\mathbf{F}_i(\mathbf{x})\}_{i=1}^N$  from the  $N$  source views to capture the global consistency information. Each image feature  $\mathbf{F}_i(\mathbf{x})$  is concatenated with the mean and variance together, and then fed into a tiny MLP network to generate a new feature  $\mathbf{F}_i'(\mathbf{x})$ . Subsequently, we feed the new feature  $\mathbf{F}_i'(\mathbf{x})$ , the displacement  $\Delta \mathbf{d}_i = \mathbf{x} - \mathbf{o}_i$  between the 3D point  $\mathbf{x}$  and the camera origin  $\mathbf{o}_i$  of the corresponding source view, and the trilinearly interpolated volume encoding feature  $\mathbf{M}(\mathbf{x})$  into an MLP network  $f_v$  to generate voting weights  $v_i(\mathbf{x})$ . In contrast to view-dependent color, we use only displacement for semantics since it remains consistent regardless of the viewing direction.
Furthermore, we encode the displacement information  $\Delta \mathbf{d}_i = \mathbf{x} - \mathbf{o}_i$  for voting weights prediction since nearby views should contribute more compared to the distant ones. The voting weight  $v_{i}(\mathbf{x})$  is predicted as:
$$
v _ {i} (\mathbf {x}) = f _ {v} \left(\mathbf {F} _ {i} ^ {\prime} (\mathbf {x}), \mathbf {M} (\mathbf {x}), \Delta \mathbf {d} _ {i}\right). \tag {3}
$$
The final semantics of the 3D point is obtained through a soft voting mechanism, given by:
$$
\mathbf {s} (\mathbf {x}) = \sum_ {i = 1} ^ {N} \left(\mathbf {S} _ {i} (\mathbf {x}) \cdot \exp \left(v _ {i} (\mathbf {x})\right) / \sum_ {j = 1} ^ {N} \exp \left(v _ {j} (\mathbf {x})\right)\right), \tag {4}
$$
where  $\exp (.)$  represents the exponential function.
Remarks. An alternative approach would be to directly regress  $\mathbf{s}(\mathbf{x})$  instead of predicting the voting weights. Compared to this alternative approach, predicting the voting weights makes it easier for the network to leverage the source-view semantics maps which are easier to obtain without direct 3D supervision. Our conjecture is verified experimentally by the degraded performance when we directly regress the semantics.
3.3 Semantics Rendering and Training
**Rendering.** The method presented in the previous section enables the computation of semantics  $\mathbf{s}(\mathbf{x})$  and geometry  $\sigma(\mathbf{x})$ , e.g., volume density or SDF in the continuous 3D space. The volume rendering of the semantics  $\hat{\mathbf{S}}(\mathbf{r})$  along a ray  $\mathbf{r}$  traversing the scene is approximated by the weighted sum of the semantics from the  $M$  samples along the ray, i.e.:
$$
\hat {\mathbf {S}} (\mathbf {r}) = \sum_ {m = 1} ^ {M} w \left(\mathbf {x} _ {m}\right) \cdot \mathbf {s} \left(\mathbf {x} _ {m}\right), \tag {5}
$$
where the samples from 1 to  $M$  are sorted in ascending order based on their depth values.  $\mathbf{x}_m$  represents the query point of the  $m$ -th sample along the ray  $\mathbf{r}$ .  $\mathbf{s}(\mathbf{x}_m)$  and  $w(\mathbf{x}_m)$  represent the semantics and the corresponding weight of the query sample  $\mathbf{x}_m$ , respectively. The computation of the weight  $w(\mathbf{x}_m)$  depends on the task of view synthesis or 3D segmentation (c.f. Sec. 4 for details).
Training objective. To train the model, we minimize the cross-entropy error between the rendered semantics  $\hat{\mathbf{S}} (\mathbf{r})$  and the corresponding ground truth pixel semantics  $\mathbb{S}(\mathbf{r})$ :
$$
\mathcal {L} _ {s} = - \frac {1}{| R |} \sum_ {\mathbf {r} \in R} \mathbb {S} (\mathbf {r}) \log \hat {\mathbf {S}} (\mathbf {r}), \tag {6}
$$
where  $R$  represents the set of rays included in each training batch.
4 Semantic View Synthesis and 3D Segmentation
Our proposed generalizable neural semantic field can be applied to the tasks of semantic view synthesis and 3D semantic segmentation.
4.1 Semantic View Synthesis
Semantic view synthesis aims to render the semantic maps for novel views. A naive approach for synthesizing novel semantic views first uses a generalizable NeRF to generate images of the novel views and then followed by applying 2D semantic segmentation to these synthesized images using a separate 2D semantic segmentor. In contrast to this two-stage method, our approach is a one-step synthesis that is more efficient and streamlined. In semantic view synthesis, we use the predicted geometry  $\sigma(\mathbf{x}_m)$  from Eq. 2 as the volume density. We then follow NeRF [30] to compute the weight  $w(\mathbf{x}_m)$  used in Eq. 5 by:
$$
w \left(\mathbf {x} _ {m}\right) = T \left(\mathbf {x} _ {m}\right) \left(1 - \exp (- \sigma \left(\mathbf {x} _ {m}\right))\right), \quad \text {w h e r e} T \left(\mathbf {x} _ {m}\right) = \exp \left(- \sum_ {j = 1} ^ {m - 1} \sigma \left(\mathbf {x} _ {m}\right)\right) \tag {7}
$$
is the accumulated transmittance along the ray. Once the weights  $w(\mathbf{x}_m)$  are computed, we proceed to construct and train a generalizable neural semantic field based on the approach proposed in Sec. 3.3 for rendering semantic views.
4.2 3D Semantic Segmentation
The goal of 3D semantic segmentation is to predict the semantic of each sample point in the 3D space. Consequently, the rendering of the semantic map is dependent on high-quality surfaces. To this end, we predict the geometry  $\sigma (\mathbf{x}_m)$  as a signed distance field (SDF) based on Eq. 2. Following [1], we then compute the weights  $w(\mathbf{x}_m)$  by:
$$
w \left(\mathbf {x} _ {m}\right) = \operatorname {S i g m o i d} \left(\frac {\sigma \left(\mathbf {x} _ {m}\right)}{t r}\right) \cdot \operatorname {S i g m o i d} \left(- \frac {\sigma \left(\mathbf {x} _ {m}\right)}{t r}\right), \tag {8}
$$
where  $tr$  represents the truncation distance. We set the weights of samples beyond the first truncation region to zero to account for possible multiple intersections. Subsequently, the normalized weight  $w(\mathbf{x})$  used in Eq. 5 is calculated as:
$$
w \left(\mathbf {x} _ {m}\right) = \frac {w \left(\mathbf {x} _ {m}\right)}{\sum_ {j = 1} ^ {M} w \left(\mathbf {x} _ {j}\right)}. \tag {9}
$$
In this scheme, the highest integration weight is assigned to surface points, while the weights for the points that are farther away are reduced.