SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Visibility module. While most nearby views offer valuable semantic observations for a 3D point  $\mathbf{x}$ , there exists some views that contribute negatively due to occlusion. The most straightforward approach to address occlusion is to render the depth map in the same manner as color or semantics. However, this method incurs additional computational cost. To address occlusion efficiently, we leverage the property of signed distance function (SDF): the SDF value is positive in front of the surface and negative behind it. This implies that if a source view can observe a target 3D point, the SDF values of the points along the ray transit from positive to negative at the origin of the source view to the target point (c.f.  $\mathbf{x}_1$  in Fig.2b). However, in the presence of occlusion, the value changes from positive to negative and then back to positive (c.f.  $\mathbf{x}_2$  and  $\mathbf{x}_3$  in Fig.2b). We use this property to mask out occluded views and mitigate the influence of occlusion.
We evaluate our method on the tasks of semantic view synthesis in Sec. 5.1.1 and 3D semantic segmentation in Sec. 5.1.2. Additionally, we validate the effectiveness of the proposed modules in Sec. 5.2.
Dataset and Metrics. We perform the experiments on two indoor datasets: ScanNet (V2) [13] and Replica [38]. The ScanNet dataset contains 1,613 indoor scenes with ground-truth camera poses, surface reconstructions, and semantic segmentation labels. We follow [31] and [40] on the three training/validation/test splits commonly used in previous works. The 2D semantic segmentor Mask2Former is pre-trained on the train split of ScanNet without any additional amendment. Furthermore, we follow the 20-class semantic segmentation task defined in the original ScanNet benchmark. In contrast to NeRF-based methods [53, 56] which only use the training subset of ScanNet to train and test their methods, we train and test our method on the complete dataset. Replica consists of 16 high-quality 3D scenes of apartments, rooms, and offices. We use the same 8 scenes as in [55]. While Replica originally features 100 classes, we rearranged them into 19 frequently observed semantic classes. We use the model trained on ScanNet to perform the validation on Replica. For the evaluation metrics, we use the mean of class-wise intersection over union (mIoU) and the average pixel accuracy (mAcc). For 3D semantic segmentation, we follow the evaluation procedure of Atlas [31].
Table 1: Quantitative comparison on semantic view synthesis from the validation set of ScanNet. 'Ours-100' means the model trained on 100 scenes and 'Ours-1000' trained on full training set of ScanNet. Semantic-Ray is also trained on full training set of ScanNet. 'Predict Directly' represents the method that incorporates a semantic head into the generalizable NeRF framework.
Method	Novel View	Novel Scene
mIoU (%)	mAcc (%)	mIoU (%)	mAcc (%)
Per-scene optimized	Semantic-NeRF [55]	96.8	98.2	-	-
DM-NeRF [3]	93.5	96.1	-	-
PNF [23]	94.1	97.2	-	-
Gneralizable	Mask2Former [10]	80.9	90.7	64.6	77.9
Predict Directly	50.2	62.3	39.1	54.2
Semantic-Ray [24]	-	-	56.0	-
Ours-100	93.3	96.3	47	60.8
Ours-1000	87.8	93.3	71.6	82.3
5.1 Comparisons to Baselines
5.1.1 Semantic View Synthesis
We compare with state-of-the-art per-scene optimized neural semantic field methods: Semantic-NeRF [55], Panoptic Neural Fields (PNF) [23] and DM-NeRF [3]. For each per-scene optimized method, we train a NeRF model on each scene from the train set of ScanNet, and then compute the 2D mIoU
across these scenes. Note that although PNF and DM-NeRF aim to predict panoptic segmentation, they can still predict semantic segmentation. We train our method for different number of scenes, including 100 scenes (denoted as Ours-100) and full training set (denoted as Ours-1000), to show the capacity of our model over different scenarios. For generalizable methods, we compare with Mask2Former, Semantic-Ray [24] and the method adding a semantic head to a generalizable NeRF (denoted as Predict Directly) in the same training and testing setting. For Mask2Former, IBRNet [45] is first used to render color images on novel views, and then Mask2Former is used to predict semantic results on rendered images. For Predict Directly, we use IBRNet as its generalizable NeRF framework.
As shown in Tab. 1, we outperform the baselines across ScanNet on semantic segmentation tasks. Compared with per-scene optimized methods, our method can obtain comparable performance (93.3% v.s. 93.5% 2D mIoU) on novel views. Moreover, only our methods can be generalized to novel scenes (47% 2D mIoU). Although Semantic-NeRF performs better than our method (+3.5%), it requires per-scene optimization and thus not generalizable to unseen scenes. Comparing with other generalizable methods, our method obtains the state-of-the-art performance. For example, our method significantly improves over Mask2Former by 7%, which shows our method is better and more concise. Furthermore, our method improves significantly (71.6% v.s. 56%) compared with Semantic-Ray. In addition, our method has improved tremendously (71.6% v.s. 39.1%) over the naive approach that incorporates a semantic head into the generalizable NeRF framework. This validates the effectiveness of our soft voting scheme.
Table 2: Quantitative comparison on 3D semantic segmentation from the val and test set of ScanNet. Note that 'Geometry' means using 3D geometry as input and 'Sup.' represents supervision.
Method	Val Split (mIoU)	Test Split (mIoU)	Geometry
3D Sup.	Atlas [31]	36.8	34	X
NeuralRecon [40]	37.5	35.1	X
Joint Recon-Segment [29]	-	51.5	X
PointNet++ [33]	53.5	33.9	✓
PointConv [47]	61	55.6	✓
2D Sup.	S4R [20]	36.9	-	✓
ScanComplete [15]	29.5	-	✓
Ours	58.8	55.1	X
Ours with Mesh	60.4	55.8	✓
5.1.2 3D semantic segmentation
We evaluate 3D semantic segmentation on the val and test set of ScanNet. We compare our method with approaches trained on 3D semantic supervision, i.e., Atlas [31], NeuralRecon [40], Joint Recon-Segment [29], PointConv [47], and PointNet++ [33], and approaches trained on 2D semantic supervision, i.e., S4R [20], and ScanComplete [15]. The input of these methods are different: PointConv [47], PointNet++ [33], S4R [20], and ScanComplete [15] use 3D geometry, e.g., point cloud, 3D mesh, etc, as input; Others, i.e., Atlasm NeuralRecon, and Joint Recon-Segment [29] use RGB images as input to predict 3D surfaces and use an extra semantic head to predict semantics. In Tab. 2, we use 'Geometry' to denote 3D geometry as input and 'Sup.' to represent supervision.
As shown in Tab. 2, our proposed method achieves comparable performance to the methods with 3D semantic supervision, and significantly outperforms all methods with 2D semantic supervision. For example, our approach performs significantly better than all other methods using RGB images as
Table 3: Quantitative comparison on Replica.
Method	Atlas	NeuralRecon	Ours
3D mIoU	25.4	26.2	43.8
Figure 3: Qualitative 3D semantic segmentation results on ScanNet. As we can see from the figure, our method is better than NeuralRecon, especially on the 'bed' class. In detail, NeuralRecon mistakenly classifies the bed as a table (highlight in black box).
input. The highest mIoU of the previously best method that predicts 3D surface on the ScanNet test set is  $51.5\%$  [29], which is significantly less than our results of  $55.1\%$ . More importantly, they need 3D semantic supervision while we only require 2D semantic supervision. Our approach still performs better than some of the methods (58.8% v.s. 53.5% [33]) with 3D supervision and 3D geometry as input. Our method is comparable to PointConv which uses 3D Geometry as input (60.4% v.s. 61%) when we replace SDF prediction with ground truth. Similar to our method, S4R leverages differentiable rendering as training supervision. However, S4R incorporates a semantic head into the existing NeRF framework and uses 3D geometry as input. Our approach outperforming S4R (36.9% v.s. 58.8%) proves the effectiveness of our proposed soft voting scheme. We also show qualitative comparison with NeuralRecon in Fig. 3. We can see that NeuralRecon wrongly classifies the bed as table (highlight in black box), while our approach makes the correct prediction. Tab. 3 shows the performance of our method compared with Altas [31] and NeuralRecon [40] that also use color images as input when trained on ScanNet and test on Replica. We can see that our method shows much stronger ability (43.8% v.s. 25.4%/26.2%) for cross-dataset generalization.
As shown in Tab. 4, we do ablation study to investigate the contribution of each component proposed in our method on ScanNet. We start with a baseline method predicting the semantics directly of each 3D query point. Next, we use only semantic logits to vote for semantics of each 3D query point. We then performed a series of tests where we include each component of our framework one-by-one and measured the impact on the performance. The third row shows the impact of using a probability distribution instead of semantic logits, and the fourth shows the impact of adding the visibility module. We find that each component improves the 3D segmentation IoU performance.
Table 4: Ablations of our design choices on 3D semantic segmentation.
Predict Directly	Logit	Prob.	Visibility Module	mIoU	mAcc
✓	✗	✗	✗	37.5 (+0)	47.0 (+0)
✗	✓	✗	✗	57.3 (+19.8)	70.9 (+23.9)
✗	✗	✓	✗	57.9 (+0.6)	71.2 (+0.3)
✗	✗	✓	✓	58.8 (+0.9)	72.1 (+0.9)
In this paper, we introduce a generalizable 3D segmentation framework based on implicit representation. Specifically, we propose a novel soft voting mechanism on 2D semantic information from multiple viewpoints to predict the semantics for each 3D point. Instead of using positional information, our framework utilizes multi-view image features as input and thus avoids the need to memorize scene-specific geometric and semantic details. Additionally, our framework encodes view differences to give higher importance to nearby views compared to distant ones when predicting the voting scores. We also design a visibility module to identify and discard detrimental information from occluded views. By incorporating view difference encoding and visibility prediction, our framework achieves more effective aggregation of the 2D semantic information. Experimental results demonstrate that our approach performs comparably to scene-specific methods. More importantly, our approach even surpasses existing strong supervision-based approaches with just 2D annotations.
This research work is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021).
[1] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6290–6301, June 2022.
[2] Aude Billard and Danica Kragic. Trends and challenges in robot manipulation. Science, 364(6446):eaat8414, 2019.
[3] Wang Bing, Lu Chen, and Bo Yang. Dm-nerf: 3d scene geometry decomposition and manipulation from 2d images. arXiv preprint arXiv:2208.07227, 2022.
[4] Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver. Visual navigation for mobile robots: A survey. Journal of intelligent and robotic systems, 53:263-296, 2008.
[5] Ang Cao, Chris Rockwell, and Justin Johnson. Fwd: Real-time novel view synthesis with forward warping and depth. CVPR, 2022.
[6] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124-14133, 2021.
[7] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-848, 2017.
[8] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.
[9] Shenchang Eric Chen and Lance Williams. View interpolation for image synthesis. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques, pages 279-288, 1993.
[10] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022.
[11] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.