Title: GNeSF: Generalizable Neural Semantic Fields
Venue: NeurIPS 2023
Accepted: yes
Keywords: NeRF; Semantic Segmentation; 3D vision; Scene understanding; Generalizable
Abstract: 3D scene segmentation based on neural implicit representation has emerged recently with the advantage of training only on 2D supervision. However, existing approaches still requires expensive per-scene optimization that prohibits generalization to novel scenes during inference. To circumvent this problem, we introduce a \textit{generalizable} 3D segmentation framework based on implicit representation. Specifically, our framework takes in multi-view image features and semantic maps as the inputs instead of only spatial information to avoid overfitting to scene-specific geometric and semantic information. We propose a novel soft voting mechanism to aggregate the 2D semantic information from different views for each 3D point. In addition to the image features, view difference information is also encoded in our framework to predict the voting scores. Intuitively, this allows the semantic information from nearby views to contribute more compared to distant ones. Furthermore, a visibility module is also designed to detect and filter out detrimental information from occluded views. Due to the generalizability of our proposed method, we can synthesize semantic maps or conduct 3D semantic segmentation for novel scenes with solely 2D semantic supervision. Experimental results show that our approach achieves comparable performance with scene-specific approaches. More importantly, our approach can even outperform existing strong supervision-based approaches with only 2D annotations.

GNeSF: Generalizable Neural Semantic Fields

Abstract

3D scene segmentation based on neural implicit representation has emerged recently with the advantage of training only on 2D supervision. However, existing approaches still requires expensive per-scene optimization that prohibits generalization to novel scenes during inference. To circumvent this problem, we introduce a generalizable 3D segmentation framework based on implicit representation. Specifically, our framework takes in multi-view image features and semantic maps as the inputs instead of only spatial information to avoid overfitting to scene-specific geometric and semantic information. We propose a novel soft voting mechanism to aggregate the 2D semantic information from different views for each 3D point. In addition to the image features, view difference information is also encoded in our framework to predict the voting scores. Intuitively, this allows the semantic information from nearby views to contribute more compared to distant ones. Furthermore, a visibility module is also designed to detect and filter out detrimental information from occluded views. Due to the generalizability of our proposed method, we can synthesize semantic maps or conduct 3D semantic segmentation for novel scenes with solely 2D semantic supervision. Experimental results show that our approach achieves comparable performance with scene-specific approaches. More importantly, our approach can even outperform existing strong supervisionbased approaches with only 2D annotations. Our source code is available at: https://github.com/HLinChen/GNeSF.

1 Introduction

Understanding high-level semantics of 3D scenes in digital images and videos is a fundamental research in computer vision. Several extensively studied tasks such as scene classification [22], object detection [34], and semantic segmentation [10] facilitate the extraction of semantic descriptions from RGB and other sensor data. These tasks form the foundation for applications such as visual navigation [4] and robotic interaction [2].

Several commonly used 3D representations include 3D point clouds [11, 32], voxel grids [42], and polygonal meshes [28, 18]. A direct learning of semantics on these 3D representations require large amounts of well-annotated 3D ground truths for training. Unfortunately, these 3D ground truths are often much more laborious and expensive to obtain compared to its 2D counterparts. Recently, Neural Radiance Field (NeRF) [30, 52] has emerged as a new 3D representation, which can capture intricate geometric details [41] using solely RGB images. Many researchers utilize this 3D representation to propose the concept of Neural Semantic Field [55, 17, 3], where semantics is assigned to each 3D point. However, these existing works on neural semantic fields inherited the per-scene optimization limitation of the vanilla NeRF and thus cannot generalize across novel scenes. As a result, the practicality of the approaches for semantic segmentation is severely restricted.

In view of the limitations in existing neural semantic field works, this paper addresses the challenging task of generalizable neural semantic fields. The objective is to construct a generalizable semantic field capable of producing high-quality semantic rendering from novel views and 3D semantic

segmentation using input images from unseen scenes. The generalization ability offers numerous interactive applications that provide valuable feedback to users during 3D capture without the requirement for retraining. Recently, NeSF [44] claims the ability to generate 3D semantic fields for novel scenes by utilizing 3D UNet [36] to extract semantic features from a density field obtained from a pre-trained NeRF model. However, this method lacks efficiency as it requires training a separate NeRF model for each scene before segmenting the 3D scene. Furthermore, it lacks sufficient generalizability to new scenes, thus making it suitable only for offline applications. Recently, several researchers propose generalizable NeRF methods [6, 45, 53] that demonstrate effective generalization across scenes for reconstructing radiance fields without the need for laborious per-scene optimization. A naive approach for generalizable Neural Semantic Field (NeSF) is to incorporate an additional semantic head to the generalizable NeRF framework. However, this method does not perform well for complex and realistic indoor scenes, as demonstrated in Tab. 1 and [20]. One possible reason is that semantics represents high-level information, and a simple head without direct 3D semantic supervision is insufficient for generating 3D semantic fields.

Inspired by image-based rendering (IBR) [9, 45], we propose a soft voting scheme (shown in Fig. 2a) to aggregate the 2D semantic information from different views for each 3D point for learning a Generalizable Neural Semantic Field (GNeSF). Specifically, we use a pre-trained 2D semantic segmentation model to generate 2D semantic maps for the source views and then leverage the warping, resampling, and/or blending operations of IBR to infer the semantics of a sample 3D point. Considering the unbounded nature and the lack of meaning for logit values across multiple views, we utilize a probability distribution instead of semantic logits to represent semantics across views. Our framework utilizes multi-view image features instead of position information as input to further enhance the generalizability of our model and prevent overfitting to scene-specific geometric and semantic information. In addition to the extracted image features, our framework incorporates view difference information to include the prior of higher significance for nearby views when generating the voting scores for blending source views. We then introduce a soft voting scheme to compute the semantics of each sample point as the weighted combination of the projected semantics from the source views.

Geometric information is predicted to perform volumetric rendering after obtaining semantics of 3D points along sampled ray. Specifically, we first build a feature volume by projecting each vertex of the volume to latent 2D features and aggregating these 2D features to get a density 3D feature. This feature volume is then fed into a 3D neural network to obtain a geometry encoding volume. For a sampled 3D point, we use an MLP network to infer the specific geometric representations, e.g., density or Signed Distance Function (SDF), from the interpolated features of the geometry encoding volume.

Weights of points along the ray are obtained from the predicted geometric representations. A final semantic value is computed for each ray through volume rendering by weighted summing semantics alone the ray. Due to the generalizability of our proposed method, we can synthesize semantic maps or conduct 3D semantic segmentation for novel scenes with solely 2D semantic supervision. To facilitate 3D semantic segmentation, we design a visibility module (shown in Fig. 2b) that identifies and removes occluded views for each spatial point. The remaining semantics from the visible views are then utilized to vote on the semantics of the respective point. In summary, our contributions include:

We are the first to tackle the task of generalizable neural semantic field. Our proposed GNeSF is capable of inferring novel view synthesis of semantic maps or 3D semantic segmentation of novel scenes without retraining. Furthermore, our method depends solely on 2D semantic supervision during training.

We propose a novel soft voting scheme that determines the semantics for each 3D point by considering the 2D semantics from multiple source views. A visibility module is also designed to detect and filter out detrimental information from occluded views.

Our proposed GNeSF enables synthesis of semantic maps for both novel views and novel scenes, and achieves comparable performance to per-scene optimized methods. The results on ScanNet and Replica demonstrate comparable or superior performance compared to existing methods utilizing strong 3D supervision.

2 Related Work

2.1 2D & 3D Semantic Segmentation

Traditionally, semantic segmentation has been treated as a per-pixel classification task, where FCNbased architectures [27] assign category labels to individual pixels independently. Subsequent methods have acknowledged the importance of context in achieving accurate per-pixel classification and have concentrated on developing specialized context modules [7, 8, 54] or incorporating selfattention variations [39, 46, 10]. In our work, we utilize the state-of-the-art Mask2Former [10] with a Swin-Transformer [26] backbone as our pre-trained 2D semantic segmenter. 3D semantic segmentation also has been investigated with pre-computed 3D structures such as voxel grids or point clouds [19, 32, 33, 37, 12, 14, 35], and simultaneous segmentation and 3D reconstruction from 2D images [31, 29]. Unlike these methods, our GNeSF aims to segment a dense 3D representation using only 2D inputs and semantic supervision without the need for 3D semantic annotations.

2.2 Neural Semantic Fields

Semantic-NeRF [55] is first to introduce the integration of semantics into NeRF. It showcases the fusion of noisy 2D semantic segmentations into a coherent volumetric model. This integration enhanced the accuracy of the model and enabled the synthesis of novel views with semantic masks. Subsequently, numerous studies have built upon this concept. For example, [17, 23, 21] incorporate instance modeling, and [43] encode abstract visual features that allow for post hoc derivation of semantic segmentation. NeRF-SOS [16] combines object segmentation and neural radiance fields to segment objects in complex real-world scenes using self-supervised learning. [50] proposes an objectcompositional neural radiance field. RFP [25] designs an unsupervised approach to segment objects in 3D during reconstruction using only unlabeled multi-view images of a scene. Panoptic NeRF [17] and DM-NeRF [3] are specifically designed for panoptic radiance fields for tasks such as label transfer and scene editing, respectively. However, these methods require scene-specific optimization and thus not generalizable to unseen scenes. Similar to our objective, NeSF [44] addresses the challenge of generalization to unseen scenes. Nonetheless, NeSF relies on multiple pre-trained and scene-specific optimized NeRF models for density field generation and segmentation which restricts its overall generalizability. In contrast, our GNeSF is able to infer on novel scenes directly without any further optimization.

2.3 Generalizable Neural Implicit Representation

Significant advancements have been achieved in the field of neural implicit representation [30, 48, 52]. However, these approaches heavily depend on computationally expensive per-scene optimization. To address this limitation, several new neural implicit representation methods emphasizing on generalization has emerged. These methods [6, 45, 49, 51, 5] aim to learn a representation of the radiance field from a provided set of images of novel scenes. This enables the synthesis of novel views without requiring per-scene optimization. PixelNeRF [51] and IBRNet [45] utilize volume rendering techniques to generate novel view images by employing warped features from nearby reference images. In addition to novel synthesis, MonoNeuralFusion [56] introduces a novel neural implicit scene representation with volume rendering for high-fidelity generalizable 3D scene reconstruction from monocular videos. Inspired by these generalizable methods, we takes in multi-view image feature as the input instead of the position information to enhance the generalizability of our model. Semantic knowledge obtained from 2D vision is then aggregated to infer 3D semantics by the proposed soft voting scheme.

3 Our GNeSF: Generalizable Neural Semantic Fields

We propose GNeSF to achieve generalizable semantic segmentation on neural implicit representation. As illustrated in Fig. 1, our GNeSF consists of three main components: 1) Feature extraction and semantic prediction from multiple source views; 2) Geometry and semantics prediction for sampled 3D points; 3) Semantic map prediction with volume rendering.

Encoding

Volume

2D

CNN

3D

Enc.

MVS

FC

Feature

Volume

FC

M2Former

…

…

…

Query

𝒙

Query

𝝅

𝒊

(𝒙)

…

Nearby

Image

Nearby

Image

Target

Image

𝝅

𝒊

(𝒙)

3D

point

𝒙

𝝅

𝒊

(𝒙)

Soft Voting

⨁

Geometry

𝝈(𝒙)

Weight

𝒘(𝒙)

Voting Weight

𝒗

𝒊

(𝒙)

2D Semantics

𝐒

𝒊

(𝒙)

Queried 2D Features

𝐅

𝐢

(𝒙)

2D Feature map

𝐅

𝐢

𝒊#𝟏

𝑵

Queried 3D

Features

𝐌(𝒙)

Volume rendering

𝐒 ' (𝐫)

3D Semantics s

(𝒙)

Rendered

Semantics

Concatenate

Ray

𝐫

3.1 Source-View Image Features and Semantics Map

Inspired by existing generalizable neural implicit representations [6, 45], our framework leverages image features from source-view images for generalization to unseen scenes during inference. We extract an image feature map F i ∈ R H i × W i × d for each source image I i ∈ [0 , 1] H i × W i × 3 with a U-Net-based convolutional neural network, where H i × W i and d denotes the dimensions of the image and the feature maps, respectively. Concurrently, we use a pre-trained mask2former [10] to predict the semantic maps S i ∈ R H i × W i × c of each source image I i , where c represents the number of categories.

3.2 3D Point Geometry and Semantics Predictions

Using the source-view image features and semantics maps, we predict the geometry σ ( x ) and semantics s ( x ) for each sampled 3D point x ∈ R 3 . We first build a geometry encoding volume M based on the image feature map F i and then predict the geometry σ ( x ) using an MLP. Concurrently, we design a soft voting scheme (shown in Fig. 2a) based on the 2D semantic observations S i ∈ R H i × W i × c to predict the semantics s ( x ) .

3.2.1 Geometry Encoding Volume

To construct the geometry encoding volume M, we project each vertex k ∈ R 3 of the volume to the image feature maps F i and obtain its image features F i ( π i ( k )) by interpolation, where π i ( k ) denotes the projected pixel location of k on the feature map F i . For simplicity, we abbreviate F i ( π i ( k )) as F i ( k ) . We then compute the mean µ ∈ R d and variance v ∈ R d of the projected image features { F i ( k ) } N i =1 from the N source views to capture the global information. The mean µ and variance v are concatenated to build a feature volume B . Intuitively, the mean image feature provides cues for the appearance information, and the variance for geometry reasoning. Finally, we use a 3D neural network H g to aggregate the feature volume B to obtain the geometry encoding volume M . Formally, the whole process can be expressed by:

where [ · , · ] represents feature concatenation, and Mean ( · ) and Var ( · ) are the averaging and variance operations. For a sampled 3D point x , we use an MLP network f θ to predict the geometry σ ( x ) based on the interpolated features of the geometry encoding volume M ( x ) , i.e. :

SOFA

DESK

CHAIR

BED

Prob

Class

Prob

Class

Prob

Class

Prob

Class

Prob

Class

0.3

0.5

0.2

(a) Soft Voting Scheme

SDF

Ray

Prob

Class

2D Semantic

Observation

Object

𝒙

!

𝒙

"

+1

-1

0

𝒙

#

(b) Visibility Module

Note that geometry σ ( x ) , e.g., volume density or Signed Distance Function (SDF) is represented differently in different neural implicit representations. More details are provided in Sec. 4.

3.2.2 Semantics Soft Voting

Given a query point location x ∈ R 3 along a ray r ∈ R 3 , we project x onto N nearby source views using the respective camera parameters. Image features { F i ( x ) } N i =1 ∈ R d and semantics maps { S i ( x ) } N i =1 ∈ [0 , 1] k are then sampled at the projected pixel locations over the N source views through bilinear interpolation on the image feature F i and semantic maps S i , respectively. We propose a soft voting scheme to predict the semantics s ( x ) of a sampled 3D point x based on the 2D projected image features { F i ( x ) } N i =1 and semantics maps { S i ( x ) } N i =1 . Specifically, the 2D projected image features { F i ( x ) } N i =1 are used to predict voting weights { v i } N i =1 for the corresponding semantics maps { S i ( x ) } N i =1 in the source views.

We first calculate the mean and variance of the image features { F i ( x ) } N i =1 from the N source views to capture the global consistency information. Each image feature F i ( x ) is concatenated with the mean and variance together, and then fed into a tiny MLP network to generate a new feature F ′ i ( x ) . Subsequently, we feed the new feature F ′ i ( x ) , the displacement ∆ d i = x -o i between the 3D point x and the camera origin o i of the corresponding source view, and the trilinearly interpolated volume encoding feature M ( x ) into an MLP network f v to generate voting weights v i ( x ) . In contrast to view-dependent color, we use only displacement for semantics since it remains consistent regardless of the viewing direction.

Furthermore, we encode the displacement information ∆ d i = x -o i for voting weights prediction since nearby views should contribute more compared to the distant ones. The voting weight v i ( x ) is predicted as:

The final semantics of the 3D point is obtained through a soft voting mechanism, given by:

where exp( . ) represents the exponential function.

Remarks. An alternative approach would be to directly regress s ( x ) instead of predicting the voting weights. Compared to this alternative approach, predicting the voting weights makes it easier for the network to leverage the source-view semantics maps which are easier to obtain without direct 3D supervision. Our conjecture is verified experimentally by the degraded performance when we directly regress the semantics.

3.3 Semantics Rendering and Training

Rendering. The method presented in the previous section enables the computation of semantics s ( x ) and geometry σ ( x ) , e.g., volume density or SDF in the continuous 3D space. The volume rendering of the semantics ˆ S ( r ) along a ray r traversing the scene is approximated by the weighted sum of the semantics from the M samples along the ray, i.e. :

where the samples from 1 to M are sorted in ascending order based on their depth values. x m represents the query point of the m -th sample along the ray r . s ( x m ) and w ( x m ) represent the semantics and the corresponding weight of the query sample x m , respectively. The computation of the weight w ( x m ) depends on the task of view synthesis or 3D segmentation ( c.f. Sec. 4 for details).

Training objective. To train the model, we minimize the cross-entropy error between the rendered semantics ˆ S ( r ) and the corresponding ground truth pixel semantics S ( r ) .:

where R represents the set of rays included in each training batch.

4 Semantic View Synthesis and 3D Segmentation

Our proposed generalizable neural semantic field can be applied to the tasks of semantic view synthesis and 3D semantic segmentation.

4.1 Semantic View Synthesis

Semantic view synthesis aims to render the semantic maps for novel views. A naive approach for synthesizing novel semantic views first uses a generalizable NeRF to generate images of the novel views and then followed by applying 2D semantic segmentation to these synthesized images using a separate 2D semantic segmentor. In contrast to this two-stage method, our approach is a one-step synthesis that is more efficient and streamlined. In semantic view synthesis, we use the predicted geometry σ ( x m ) from Eq. 2 as the volume density. We then follow NeRF [30] to compute the weight w ( x m ) used in Eq. 5 by:

is the accumulated transmittance along the ray. Once the weights w ( x m ) are computed, we proceed to construct and train a generalizable neural semantic field based on the approach proposed in Sec. 3.3 for rendering semantic views.

4.2 3D Semantic Segmentation

The goal of 3D semantic segmentation is to predict the semantic of each sample point in the 3D space. Consequently, the rendering of the semantic map is dependent on high-quality surfaces. To this end, we predict the geometry σ ( x m ) as a signed distance field (SDF) based on Eq. 2. Following [1], we then compute the weights w ( x m ) by:

where tr represents the truncation distance. We set the weights of samples beyond the first truncation region to zero to account for possible multiple intersections. Subsequently, the normalized weight w ( x ) used in Eq. 5 is calculated as:

In this scheme, the highest integration weight is assigned to surface points, while the weights for the points that are farther away are reduced.

Visibility module. While most nearby views offer valuable semantic observations for a 3D point x , there exists some views that contribute negatively due to occlusion. The most straightforward approach to address occlusion is to render the depth map in the same manner as color or semantics. However, this method incurs additional computational cost. To address occlusion efficiently, we leverage the property of signed distance function (SDF): the SDF value is positive in front of the surface and negative behind it. This implies that if a source view can observe a target 3D point, the SDF values of the points along the ray transit from positive to negative at the origin of the source view to the target point ( c.f. x 1 in Fig.2b). However, in the presence of occlusion, the value changes from positive to negative and then back to positive ( c.f. x 2 and x 3 in Fig.2b). We use this property to mask out occluded views and mitigate the influence of occlusion.

5 Experiments

We evaluate our method on the tasks of semantic view synthesis in Sec. 5.1.1 and 3D semantic segmentation in Sec. 5.1.2. Additionally, we validate the effectiveness of the proposed modules in Sec. 5.2.

Dataset and Metrics. We perform the experiments on two indoor datasets: ScanNet (V2) [13] and Replica [38]. The ScanNet dataset contains 1 , 613 indoor scenes with ground-truth camera poses, surface reconstructions, and semantic segmentation labels. We follow [31] and [40] on the three training/validation/test splits commonly used in previous works. The 2D semantic segmentor Mask2Former is pre-trained on the train split of ScanNet without any additional amendment. Furthermore, we follow the 20 -class semantic segmentation task defined in the original ScanNet benchmark. In contrast to NeRF-based methods [53, 56] which only use the training subset of ScanNet to train and test their methods, we train and test our method on the complete dataset. Replica consists of 16 high-quality 3D scenes of apartments, rooms, and offices. We use the same 8 scenes as in [55]. While Replica originally features 100 classes, we rearranged them into 19 frequently observed semantic classes. We use the model trained on ScanNet to perform the validation on Replica. For the evaluation metrics, we use the mean of class-wise intersection over union (mIoU) and the average pixel accuracy (mAcc). For 3D semantic segmentation, we follow the evaluation procedure of Atlas [31].

5.1 Comparisons to Baselines

5.1.1 Semantic View Synthesis

We compare with state-of-the-art per-scene optimized neural semantic field methods: Semantic-NeRF [55], Panoptic Neural Fields (PNF) [23] and DM-NeRF [3]. For each per-scene optimized method, we train a NeRF model on each scene from the train set of ScanNet, and then compute the 2D mIoU

across these scenes. Note that although PNF and DM-NeRF aim to predict panoptic segmentation, they can still predict semantic segmentation. We train our method for different number of scenes, including 100 scenes (denoted as Ours-100) and full training set (denoted as Ours-1000), to show the capacity of our model over different scenarios. For generalizable methods, we compare with Mask2Former, Sematic-Ray [24] and the method adding a semantic head to a generalizable NeRF (denoted as Predict Directly) in the same training and testing setting. For Mask2Former, IBRNet [45] is first used to render color images on novel views, and then Mask2Former is used to predict semantic results on rendered images. For Predict Directly, we use IBRNet as its generalizable NeRF framework.

As shown in Tab. 1, we outperform the baselines across ScanNet on semantic segmentation tasks. Compared with per-scene optimized methods, our method can obtain comparable performance ( 93 . 3% v.s. 93 . 5% 2D mIoU) on novel views. Moreover, only our methods can be generalized to novel scenes ( 47% 2D mIoU). Although Semantic-NeRF performs better than our method ( +3 . 5% ), it requires per-scene optimization and thus not generalizable to unseen scenes. Comparing with other generalizable methods, our method obtain the state-of-the-art performance. For example, our method significantly improves over Mask2Former by 7% , which shows our method is better and more concise. Furthermore, our improves significantly ( 71 . 6% v.s. 56% ) compared with Sematic-Ray. In addition, our method has improved tremendously ( 71 . 6% v.s. 39 . 1% ) over the naive approach that incorporates a semantic head into the generalizable NeRF framework. This validates the effectiveness of our soft voting scheme.

5.1.2 3D semantic segmentation

We evaluate 3D semantic segmentation on the val and test set of ScanNet. We compare our method with approaches trained on 3D semantic supervision, i.e. , Atlas [31], NeuralRecon [40], Joint Recon-Segment [29], PointConv [47], and PointNet++ [33], and approaches trained on 2D semantic supervision, i.e. , S4R [20], and ScanComplete [15]. The input of these methods are different: PointConv [47], PointNet++ [33], S4R [20], and ScanComplete [15] use 3D geometry, e.g., point cloud, 3D mesh, etc, as input; Others, i.e. , Atlasm NeuralRecon, and Joint Recon-Segment [29] use RGB images as input to predict 3D surfaces and use an extra semantic head to predict semantics. In Tab. 2, we use 'Geometry' to denote 3D geometry as input and 'Sup.' to represent supervision.

As shown in Tab. 2 , our proposed method achieves comparable performance to the methods with 3D semantic supervision, and significantly outperforms all methods with 2D semantic supervision. For example, our approach performs significantly better than all other methods using RGB images as

wall

floor

cabinet

chair

sofa

table

door

window

bookshelf

desk

refrigerator

bed

picture

counter

curtain

shower curtain

toilet

sink

bathtub

otherfurniture

ignored

Ours

NeuralRecon

GT

input. The highest mIoU of the previously best method that predicts 3D surface on the ScanNet test set is 51 . 5% [29], which is significantly less than our results of 55 . 1% . More importantly, they need 3D semantic supervision while we only require 2D semantic supervision. Our approach still performs better than some of the methods ( 58 . 8% v.s. 53 . 5% [33]) with 3D supervision and 3D geometry as input. Our method is comparable to PointConv which uses 3D Geometry as input ( 60 . 4% v.s. 61% ) when we replace SDF prediction with ground truth. Similar to our method, S4R leverages differentiable rendering as training supervision. However, S4R incorporates a semantic head into the existing NeRF framework and uses 3D geometry as input. Our approach outperforming S4R ( 36 . 9% v.s. 58 . 8% ) proves the effectiveness of our proposed soft voting scheme. We also show qualitative comparison with NeuralRecon in Fig. 3. We can see that NeuralRecon wrongly classifies the bed as table (highlight in black box), while our approach makes the correct prediction. Tab. 3 shows the performance of our method compared with Altas [31] and NeuralRecon [40] that also use color images as input when trained on ScanNet and test on Replica. We can see that our method shows much stronger ability ( 43 . 8% v.s. 25 . 4% / 26 . 2% ) for cross-dataset generalization.

5.2 Ablation Studies

As shown in Tab. 4, we do ablation study to investigate the contribution of each component proposed in our method on ScanNet. We start with a baseline method predicting the semantics directly of each 3D query point. Next, we use only semantic logits to vote for semantics of each 3D query point. We then performed a series of tests where we include each component of our framework one-by-one and measured the impact on the performance. The third row shows the impact of using a probability distribution instead of semantic logits, and the fourth shows the impact of adding the visibility module. We find that each component improves the 3D segmentation IoU performance.

6 Conclusion

In this paper, we introduce a generalizable 3D segmentation framework based on implicit representation. Specifically, we propose a novel soft voting mechanism on 2D semantic information from multiple viewpoints to predict the semantics for each 3D point. Instead of using positional information, our framework utilizes multi-view image features as input and thus avoids the need to memorize scene-specific geometric and semantic details. Additionally, our framework encodes view differences to give higher importance to nearby views compared to distant ones when predicting the voting scores. We also design a visibility module to identify and discard detrimental information from occluded views. By incorporating view difference encoding and visibility prediction, our framework achieves more effective aggregation of the 2D semantic information. Experimental results demonstrate that our approach performs comparably to scene-specific methods. More importantly, our approach even surpasses existing strong supervision-based approaches with just 2D annotations.

References

Dejan Azinovi´ c, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6290-6301, June 2022.

Aude Billard and Danica Kragic. Trends and challenges in robot manipulation. Science , 364(6446):eaat8414, 2019.

Wang Bing, Lu Chen, and Bo Yang. Dm-nerf: 3d scene geometry decomposition and manipulation from 2d images. arXiv preprint arXiv:2208.07227 , 2022.

Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver. Visual navigation for mobile robots: A survey. Journal of intelligent and robotic systems , 53:263-296, 2008.

Ang Cao, Chris Rockwell, and Justin Johnson. Fwd: Real-time novel view synthesis with forward warping and depth. CVPR , 2022.

Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1412414133, 2021.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Shenchang Eric Chen and Lance Williams. View interpolation for image synthesis. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques , pages 279-288, 1993.

Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022.

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3075-3084, 2019.

Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8958-8966, 2019.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017.

Angela Dai and Matthias Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 452-468, 2018.

Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias Nießner. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4578-4587, 2018.

Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776 , 2022.

Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In International Conference on 3D Vision (3DV) , 2022.

Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshcnn: A network with an edge. In Proceedings of the IEEE International Conference on Computer Vision , pages 9365-9374, 2019.

Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongbo Fu, and Chiew-lan Tai. Jsenet: Joint semantic segmentation and edge detection network for 3d point clouds. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16 , pages 222-239. Springer, 2020.

Junwen Huang, Alexey Artemorv, Yujin Chen, Shuaifeng Zhi, Kai Xu, and Matthias Niessner. S4r: Self-supervised semantic scene reconstruction from rgb-d scans. arXiv preprint arXiv:2302.03640 , 2023.

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems , volume 35, 2022.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , volume 25, pages 1097-1105, 2012.

Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12871-12881, 2022.

Fangfu Liu, Chubin Zhang, Yu Zheng, and Yueqi Duan. Semantic ray: Learning a generalizable semantic field with cross-reprojection attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Xinhang Liu, Jiaben Chen, Huai Yu, Yu-Wing Tai, and Chi-Keung Tang. Unsupervised multiview object segmentation using radiance field propagation. Advances in Neural Information Processing Systems , 35:17730-17743, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE International Conference on Computer Vision Workshops , pages 37-45, 2015.

Davide Menini, Suryansh Kumar, Martin R Oswald, Erik Sandström, Cristian Sminchisescu, and Luc Van Gool. A real-time online learning framework for joint 3d reconstruction and semantic segmentation of indoor scenes. IEEE Robotics and Automation Letters , 7(2):1332-1339, 2021.

Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision , pages 405-421. Springer, 2020.

Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-end 3d scene reconstruction from posed images. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16 , pages 414-431. Springer, 2020.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 652-660, 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition , pages 779-788, 2016.

Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3577-3586, 2017.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10529-10538, 2020.

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 , 2019.

Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7262-7272, 2021.

Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Realtime coherent 3d reconstruction from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15598-15607, 2021.

Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. 2021.

Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 4489-4497, 2015.

Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3D distillation of self-supervised 2D image representations. In Proceedings of the International Conference on 3D Vision (3DV) , 2022.

Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes, 2021.

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR , 2021.

Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7794-7803, 2018.

Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 9621-9630, 2019.

Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Citynerf: Building nerf at city scale. arXiv preprint arXiv:2112.05504 , 2021.

Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. arXiv preprint arXiv:2201.08845 , 2022.

Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13779-13788, 2021.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In CVPR , 2021.

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radiance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5449-5458, 2022.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2881-2890, 2017.

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. 2021.

Zi-Xin Zou, Shi-Sheng Huang, Yan-Pei Cao, Tai-Jiang Mu, Ying Shan, and Hongbo Fu. Mononeuralfusion: Online monocular neural 3d reconstruction with geometric priors. arXiv preprint arXiv:2209.15153 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper combines \"generalisable\" neural radiance fields and semantic radiance fields. It does so in a relatively straightforward manner, borrowing and combining ideas in these two areas, and contributing a few technical advances on top. It demonstrates good segmentation accuracy both for producing the 2D segmentation of new views and to extract 3D segmentations of the underlying scene geometry.\n\nSTRENGTHS:\nThe paper combines the benefits of \"generalisable neural fields\" and semantic neural fields, essentially inheriting the advantages of both. Compared to prior generalisable neural fields, it adds a semantic dimension, which allows to fuse 2D segmentations into the 3D reconstruction. Compared to semantic neural fields that use test-time optimisation, it can in principle be faster and use a smaller number of images.\n\nWEAKNESSES:\n*   Some key aspects of the empirical validation seem to be left untested. The main point of a \"generalisable\" radiance field compared to a radiance field optimised at test time is to be either (1) faster and/or (2) less data hungry. However, I do not believe the experiments test any of this. Table 1 shows that the proposed \"generalisable\" field  is comparable (but worse) than semantic NeRF (optimisation based) on the *training scenes*. In a training phase, the \"generalisable\" field cannot should not provide any advantage in terms of speed or data economy compared to just fitting individual scenes.\n\n    The same Table 1 shows generalisation to new scenes, which would be the main point of the \"generalisable\" field, but this experiment:\n       1. does not compare the accuracy obtained on the new scenes to performing test-time optimisation on the same scenes;\n       2. It does not show whether there is a speed advantage compared to test-time optimisation;\n       3. It does not show if there is a dat economy advantage compared to test-time optimisation.\n    In short, we only know that the generalisable field works to some extend on new scenes, but we have no idea how that compare to test-time optimisation on those scenes, which would be the obvious alternative.\n\n* In Table 1, Ours-100 performs *better* than Ours-1000 (using 10x more data) on the training scenes. This is not entirely unexpected, but it strongly suggests that the \"generalisable\" neural fields take a significant hit compared to performing per-scene optimisation. This makes it even more important to check this gap on the new scenes. At the very least, the authors should discuss this aspect of the table in some depth.\n\n* line 272: The method applied on new scenes is found to outperform Mask2Former. If I understand the method correctly, thought, it is based on fusing/warping/reweighing the Mask2Former predictions using Eq. (5), even at test time. It is not the model by itself that outperforms Mask2Former, but it is the combination of the model with Mask2Former which is better than Mask2Former alone. It may be a good idea to reinforce this point at line 272.\n\n* The comparison in Table 2 is, for the most part, indicative. It compares different methods which take different kinds of  inputs and are supervised in very different ways. For example, it sounds impressive that this method can be competitive with ScanComplete (2018) that takes as input 3D ground truth, but then again the latter performs segmentation based on an (incomplete) LiDAR point cloud, whereas this method uses Mask2Former segmentations form 2D images of the scene -- hardly comparable.\n\n* It is unclear how the model is trained. The authors do not mention any appearance / RGB loss, so it seems that all the various networks are trained purely with an objective function involving the prediction of segmentation masks. Is this the case, or do you also employ an appearance loss as in most prior \"generalisable\" radiance fields?\n\n* The goal of the paper, namely combining \"generalisable\" and semantic neural fields, is a bit incremental. There are elements of novelty in the specific technical details of the combination, but there aren't big surprises.\n\n* Minor issues:\n   * line154: the \"variance\" $\\mathbf{v}$ of the feature vectors $\\mathbf{F}_i$ should be a matrix, not a vector of the same dimension. Do you mean that this is the diagonal of the covariance matrix?\n   * line 156: why do you call $\\mathbf{B}$ a *cost volume*? These are primarily just features (the only thing related to the idea of cost is the variance term, the computation of which could be weakly interpreted as comparing features from different views).\n   * line 178: $\\Delta \\mathbf{d}_i$ is a vector, not a distance.\n\nQUESTIONS:\n* Is the \"generalisable\" radiance field actually faster or less data-hungry than test-time optimisation when applied to novel scenes at test time?\n\n* The \"generalisable\" radiance field seems to take a significant hit in accuracy when applied to new scenes at test time. How does the accuracy compared to performing test-time optimisation (i.e., semantic NerF) on those?\n\nLIMITATIONS:\n* There is no section dedicated to discuss limitations (even though the latter is recommended in the guidelines). More importantly, I could not find any substantive discussion of the limitations at all.\n\n* There is no discussion of societal impact, but I don't believe this is required for this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

comments: SUMMARY:
This paper introduces a generalizable 3D scene segmentation framework based on neural implicit representation. The proposed framework takes multi-view image features and semantic maps as inputs to reconstruct 3D geometric and semantic information.  The generalizability of the approach enables the synthesis of semantic maps or conducting 3D semantic segmentation for novel scenes with only 2D semantic supervision. Experimental results demonstrate comparable performance with scene-specific approaches and  outperform existing the methods using only 2D annotations.

STRENGTHS:
1. This method is novel, and similar approaches have not been proposed in previous papers.
2. This task is useful and also quite innovative.
3. This paper's experiments are comprehensive and thorough.

WEAKNESSES:
1. The article lacks clear explanations for many details, such as the detailed network structure and training strategy. This leads to poor reproducibility.
2. The article claims that only 2D supervision is required, but it does not explain how the SDF (signed distance function) is learned. If following the training pipeline in NeuralRecon or Mononeuralfusion, then this claim of "only needing 2D supervision" is incorrect.

QUESTIONS:
1. Can the network learn geometry information solely based on equation (6)? If so, why? Is there an intuitive way to explain this?
2. Why does Ours-1000 perform worse than Ours-100?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper, titled GNeSF, focuses on tackling the challenge of generalizable semantic segmentation using NeRF representation for both novel segmentation synthesis and 3D segmentation. The authors introduce several key contributions in this regard. Firstly, they propose utilizing a cost volume to construct a comprehensive global feature volume. Secondly, a pre-trained 2D segmentor is employed to extract semantic information from the source views. Finally, a soft voting mechanism is introduced to predict the ultimate segmentation label for the target pixel. These approaches collectively enhance the overall effectiveness and robustness of the segmentation process.\n\nSTRENGTHS:\n1. The authors tackle a crucial problem in 3D scene understanding by extending the 2D segmentation map into the 3D space, enabling segmentation for any viewpoint.\n2. The framework design presented by the authors appears logical and well-founded. Leveraging the cost volume proves to be an effective approach for encoding comprehensive scene information, while the utilization of a pre-trained 2D model allows for accurate inference of per-view semantics.\n3. The introduced voting scheme successfully addresses the challenge of visibility when assigning labels to new pixels. Notably, the experimental results demonstrate a noteworthy improvement compared to previous methods, further validating its efficacy.\n\nWEAKNESSES:\n1. Lack of references: The authors utilize the cost volume as a crucial representation for encoding 3D scene information. However, they do not provide explicit details regarding the construction process, nor do they reference important relevant papers in the deep learning era [1][2]. Furthermore, the survey conducted on the \"Neural Semantic Field\" appears insufficient, as it fails to mention several per-scene segmentation methods based on NeRF representation [3][4][5].\n\n2. Missing technical details: The meaning of \"each vertex\" mentioned in line 151 is unclear. Did the authors explicitly construct scene volume grids by determining the near and far plane for each view? Additionally, the process of interpolating the semantic logits (line 170) is not adequately explained. Furthermore, it is unclear why the authors utilize the distance between each point x and its camera origin (line 178). Given that this is an important contribution of the proposed method, it is highly recommended to highlight the motivation for using \\delta_d at the beginning of this section.\n\n3. Experimental results: It is unclear why the novel view synthesis quality decreases from 93.3 to 87.8 in Table 1 when including more scenes for training. Could you clarify if the term \"novel view\" refers to evaluating the training scenes while leaving several views for evaluation?\n\n[1] Mvsnet: Depth inference for unstructured multi-view stereo\n[2] Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching\n[3] Nerf-sos: Any-view self-supervised object segmentation on complex scenes\n[4] Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering\n[5] Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation\n\nQUESTIONS:\nThe authors primarily focus on showcasing the results of semantic segmentation in their work. However, it is equally important to assess the quality of view synthesis (color image) when rendering novel views. It would be beneficial if the authors could compare their approach with several representative generalizable NeRF methods on the Scannet dataset. I appreciate the experimental setup and would be inclined to give a higher score if the raised concerns are adequately addressed.\n\nLIMITATIONS:\nSee the weakness and quations above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed GNeSF, a generalizable neural semantic fields by incorporating contextual and semantic prior information via a U-shaped CNN and a pre-trained segmentor. Combing multi-view fused image-level features as well as soft weighting using multi-view softmax probabilities, the overall pipeline could be trained on diverse scenes and then generalize to unseen scenes with promising performance.\n\nSTRENGTHS:\n- This paper proposes a valid extension to enable a generalizable NeSF model by incorporating deep image features as well as semantic priors of multiple RGB images.\n- Promising performance on ScanNet and Replica datasets on 2D/3D shows the effectiveness of the methods. Experiments are relative adequate.\n\nWEAKNESSES:\n- Contributions and novelties.\n  -  As far as I know, Semantic-Ray [1] from CVPR2023 has already made attempt on learning a generalizable semantic NeRF model via attention, which shares similar spirits to GNeSF. What is the main difference?\nMy main concerns comes from the lack of necessary comparison and discussions to related baselines such as Semantic-Ray [1] and other generalizable NeRF (NeuRay, MVSNeRF) with a semantic head in order to make the distinction clear.\n\n  - Similarly, the discussion of using logits vs. prob. has also been somewhat investigated in PanopticLifting [2] from CVPR2023, thus further weakens the contributions.\n\n- As to numerical experiments, several necessary information is missing in both main paper and supplement and is a bit hard for readers to really judge the strict fairness of comparison.\n   - Supervision of using predicted multi-channel prob tensors vs. hard-labels for baselines. If I understand correctly, multi-channel softmax probabilities are predicted per view to serve for soft voting. For baseline methods, are these multi-channel prob tensors also used for training like [2] or only hard discrete labels are used for training baselines which lost categorical information across classes? \n  - How many nearby views are used for building the cost volumes and how does the number of views affect the final performance?\n  - I also wonder what is the specific dimension/resolution of the built cost volume? As to cover the whole scene, I assume the cost volume owns a large spatial dimension and may make the training costly? Its dimension will affect the represented level of details of deep features.\n\n- Ablations\n  - It seems that the mentioned design choices contributes quite marginally to the final performance as shown in Table4\n  - In Section 3.2, authors discusses the motivation why soft voting is used instead of directly regressing semantics. It would be good if related results could be shown in ablations (like Table4).\n\n\nOther minor comments which need further clarify:\n - It would be good to show the training expenses of GNeSF.\n- What is exactly the adopted generalizable NeRF in the experiments (i.e., Predict Directly)?\n- How to evaluate 3D segmentation as there are discrepancy between reconstructed mesh and GT mesh. Is the same procedure in Atlas used?\n\n\n\n\n[1]Semantic Ray: Learning a Generalizable Semantic Field with Cross-Reprojection Attention CVPR 2023\n\n[2]Panoptic Lifting for 3D Scene Understanding with Neural Fields CVPR 2023\n\nQUESTIONS:\nPlease see the comments in the weaknesses section above. My main concern comes from missing comparison to closely related work like Semantic-Ray  and lack explanations of several key components, which makes the technical contribution not clear in my point of view.\n\nLIMITATIONS:\nThere is no clear discussion of limitations and potential negative societal impact in the main paper and supplement.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generalizable 3D segmentation framework. The novel components include (a) a soft voting mechanism that incorporates image features, view difference aggregate the 2D semantic information for 2D views to 3D, (b) a visibility module to avoid being affected by occluded views. With only 2D supervision, this method allows for semantic maps synthesis and 3D semantic segmentation for novel 16 scenes. The method is also strongly supported by experiments.\n\nSTRENGTHS:\n1. The ability to generalize is very important in many tasks for efficiency. This paper proposes a generalizable neural semantic field, which shows significant improvements over prior work on unseen scenes. \n\n2. Compared to prior work that uses 3D supervision, this method only needs 2D semantic supervision during training and can be very helpful for 3D segmentation tasks whose annotations are very costly to obtain.\n\n3. Despite being a bit complex, the proposed model includes a few novel and thoughtful components, which may be in helpful to inspire the community.\n\nWEAKNESSES:\n1. Not enough intuition is given to explain the qualitative results. Fig. 3 shows that NeuralRecon predicts worse than the proposed method on bed. However, the authors didn't explain why this can happen (which design choice has potentially helped with this)? In the results shown in supplementary Fig. 2, it seems the proposed method has many dotted predictions (eg, red dots on the walls in the second row), what does this mean? Does it mean the proposed method may have some problem predicting the doors on the wall?\n\n2. The clarity of the figures can be improved. For example, \"MVS\" in Fig. 1 is never explained in the text. The names of the features and neural networks in Fig. 1 sometimes are not exactly the same as the names used in the text, making it a bit hard for the readers to understand.\n\nQUESTIONS:\nTab. 1 shows that ours-1000 achieves a significant improvement over ours-100 on novel scenes. Have the authors tried on even larger set of training scenes? Could the authors provide some insights on when the marginal gain on expanding the training set will be negligible? How many scenes are Mask2Former and Predict Directly trained on?\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations or potential negative societal impact. The authors could show some failures cases that indicate some common patterns where the proposed method wouldl fail on.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}