GNeSF: Generalizable Neural Semantic Fields
Authors: Hanlin Chen
Chen Li
Mengqi Guo
Zhiwen Yan
Gim Hee Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: NeRF; Semantic Segmentation; 3D vision; Scene understanding; Generalizable

Abstract: 3D scene segmentation based on neural implicit representation has emerged recently with the advantage of training only on 2D supervision. However, existing approaches still requires expensive per-scene optimization that prohibits generalization to novel scenes during inference. To circumvent this problem, we introduce a \textit{generalizable} 3D segmentation framework based on implicit representation. Specifically, our framework takes in multi-view image features and semantic maps as the inputs instead of only spatial information to avoid overfitting to scene-specific geometric and semantic information. We propose a novel soft voting mechanism to aggregate the 2D semantic information from different views for each 3D point. In addition to the image features, view difference information is also encoded in our framework to predict the voting scores. Intuitively, this allows the semantic information from nearby views to contribute more compared to distant ones. Furthermore, a visibility module is also designed to detect and filter out detrimental information from occluded views. Due to the generalizability of our proposed method, we can synthesize semantic maps or conduct 3D semantic segmentation for novel scenes with solely 2D semantic supervision. Experimental results show that our approach achieves comparable performance with scene-specific approaches. More importantly, our approach can even outperform existing strong supervision-based approaches with only 2D annotations.

GNeSF: Generalizable Neural Semantic Fields
Hanlin Chen Chen Li Mengqi Guo Zhiwen Yan Gim Hee Lee Department of Computer Science, National University of Singapore Chanlin in. chen, gimhee 2comp mus. edu. sg
Abstract
3D scene segmentation based on neural implicit representation has emerged re- cently with the advantage of training only on 2D supervision. However, existing approaches still requires expensive per-scene optimization that prohibits generaliza- tion to novel scenes during inference. To circumvent this problem, we introduce a generalizable 3D segmentation framework based on implicit representation. Specif- ically, our framework takes in multi-view image features and semantie maps as the inputs instead of only spatial information to avoid overfitting to scene-specific geometric and semantic information. We propose a novel soft voting mechanism aggregate the 2D semantic information from different views for each 3D point. addition to the image features, view difference information also encoded framework to predict the voting scores. Intuitively, this allows the semantic information from nearby views to contribute more compared to distant ones. Fur- thermore, visibility module is also designed to detect and filter out detrimental information from occluded views. Due to the generalizability of our proposed method, synthesize semantic maps conduct 3D semantic segmentation for novel scenes with solely 2D semantic supervision. Experimental results show that approach achieves comparable performance with scene-specific approaches. More importantly, our approach can even outperform existing strong supervision- based approaches with only 2D annotations. Our source code is available https///gthubbbooo///nnChChChCCt
Introduction
Understanding high-level semantics of 3D scenes in digital images and videos is fundamental research in computer vision. Several extensively studied tasks such as scene classification [22], object detection (34], and semantic segmentation [10] facilitate the extraction of semantic descriptions from RGB and other sensor data. These tasks form the foundation for applications such as visual
navigation and robotic interaction (2).
Several commonly used 3D representations include 3D point clouds || 32). voxel grids 42 and polygonal meshes 28 18] A direct learning of semantics on these 3D representations require large amounts of well-annotated 3D ground truths for training. Unfortunately. these 3D ground truths are often much more laborious and expensive to obtain compared to its 2D counterparts. Recently, Neural has emerged new 3D representation, which can capture intricate geometric details 41 using solely RGB images. Many researchers utilize this 3D representation to propose the concept of Neural Semantic Field [55] 37 where semantics is assigned each point. However, these existing works neural semantic fields inherited the per-scene optimization limitation of the vanilla NeRF and thus cannot generalize across novel scenes. As a result, the practicality of the approaches for semantic segmentation is severely restricted.
Radiance Field (NeRF) 30 52
In view of the limitations in existing neural semantic field works, this paper addresses the challenging task of generalizable neural semantic fields. The objective is to construct a generalizable semantic field capable of producing high-quality semantic rendering from novel views and 3D semantic
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

segmentation using imput images from unseen scenes. The generalization ability offers numerous interactive applications that provide valuable feedback users during 3D capture without the requirement for retraining. Recently, NeSF [44 claims the ability to generate 3D semantic fields for
novel scenes by utilizing 3D UNet [36] extract semantic features from a density field obtained
from pre-trained NeRF model. However, this method lacks efficiency as it requires training separate NeRF model for each scene before segmenting the 3D scene. Furthermore, it lacks sufficient generalizability to new scenes, thus making it suitable only for offline applications. Recently, several researchers propose generalizable NeRF methods [6. 45 [33 that demonstrate effective generalization across scenes for reconstructing radiance fields without the need for laborious per-scene optimization. A naive approach for generalizable Neural Semantic Field (NeSF) is to incorporate an additional semantic head to the generalizable NeRF framework. However, this method does not perform well for complex and realistic indoor scenes, as demonstrated Tab. T and (20). One possible reason is that semantics represents high-level information, and simple head without direct 3D semantic supervision is insufficient for generating 3D semantic fields.
Inspired by image-based rendering (IBR) 9 45 we propose soft voting scheme (shown in Fig. 2ap
to aggregate the 2D semantic information from different views for each 3D point for learning a Generalizable Neural Semantic Field (GNeSF). Specifically, we use a pre-trained 2D semantic segmentation model to generate 2D semantic maps for the source views and then leverage the warping, resampling, and/or blending operations of IBR to infer the semantics of sample 3D point. Considering the unbounded nature and the lack of meaning for logit values across multiple views, we utilize probability distribution instead of semantic logits to represent semantics across views. Our framework utilizes multi-view image features instead of position information as input to further enhance the generalizability of our model and prevent overfitting to scene-specific geometric and semantic information. In addition to the extracted image features, our framework incorporates view difference information to include the prior of higher significance for nearby views when generating the voting scores for blending source views. We then introduce a soft voting scheme to compute the semantics of each sample point as the weighted combination of the projected semantics from the source views.
Geometric information is predicted to perform volumetric rendering after obtaining semantics of 3D points along sampled ray. Specifically, first build feature volume by projecting each vertex of the volume to latent 2D features and aggregating these 2D features to get a density 3D feature. This feature volume is then fed into 3D neural network to obtain a geometry encoding volume. For a sampled 3D point, we use an MLP network to infer the specific geometric representations, e.g., density or Signed Distance Function (SDF), from the interpolated features of the geometry encoding volume.
Weights of points along the ray are obtained from the predicted geometric representations. A final semantic value is computed for each ray through volume rendering by weighted summing semantics alone the ray. Due to the generalizability of our proposed method, or conduct 3D semantic segmentation for novel scenes with solely 2D semantic supervision. To facilitate 3D semantic segmentation, we design visibility module (shown in Fig. 2b) that identifies and removes occluded views for each spatial point. The remaining semantics from the visible views then utilized to vote on the semantics of the respective point. In summary, our contributions include:
synthesize semantic maps
We are the first to tackle the task of generalizable neural semantic field. Our proposed GNeSF is capable of inferring novel view synthesis of semantic maps or 3D semantic segmentation of novel scenes without retraining. Furthermore, our method depends solely on 2D semantic supervision during training.
We propose a novel soft voting scheme that determines the semantics for each 3D point by considering the 2D semantics from multiple source views. A visibility module is also designed to detect and filter out detrimental information from occluded views.
Our proposed GNeSF enables synthesis of semantic maps for both novel views and novel scenes, and achieves comparable performance to per-scene optimized methods. The results on ScanNet and Replica demonstrate comparable or superior performance compared to existing methods utilizing strong 3D supervision.

---

2 Related Work
2.1 2D & 3D Semantic Segmentation
Traditionally, semantic segmentation has been treated as a per-pixel classification task, where FCN- based architectures [27] assign category labels to individual pixels independently. Subsequent methods have acknowledged the importance of context in achieving accurate per-pixel classification
and have concentrated developing specialized context modules 7 8 54] incorporating self-
attention variations [39 46 [0]. In our work, we utilize the state-of ofte--of-the-arr the-art Mask2Former [1O]
with Swin- Transformer [26 backbone
pre-trained 2D semantic segmenter. 3D semantic segmentation also has been investigated with pre-computed 3D structures such as voxel grids or point 35 and simultaneous segmentation and 3D reconstruction from 21)
clouds [9] 32 33
2
images 31 29 Unlike these methods, our GNeSF aims to segment a dense 3D representation using
only 2D inputs and semantic supervision without the need for 3D semantic annotations.
2.2 Neural Semantic Fields
Semantic-NeRF (55] is first to introduce the integration of semantics into NeRF. It showcases the fusion of noisy 2D semantic segmentations into a coherent volumetric model. This integration enhanced the accuracy of the model and enabled the synthesis of novel views with semantic masks. Subsequently, numerous studies have built upon this concept. For example, 17 23 21 incorporate instance modeling, and 43 encode abstract visual features that allow for post hoc derivation of semantic segmentation. NeRF-SOS [16] combines object segmentation and neural radiance fields to segment objects in complex real-world scenes using self-supervised learning. (50) proposes an object-
compositional neural radiance field. RFP (25) designs an unsupervised approach to segment objects in 3D during reconstruction using only unlabeled multi-view images of scene. Panoptic NeRF [17) and DM-NeRF 3 are specifically designed for panoptic radiance fields for tasks such as label transfer and scene editing. respectively. However, these methods require scene-specific optimization and thus not generalizable to unseen scenes. Similar to our objective, NeSF 44 addresses the challenge of generalization unseen scenes. Nonetheless, NeSF relies on multiple pre-trained and scene-spes speciffc optimized NeRF models for density field generation and segmentation which restricts its overall generalizability. In contrast, our GNeSF is able to infer on novel scenes directly without any further optimization.
2.3 Generalizable Neural Implicit Representation
Significant advancements have been achieved in the field of neural implicit representation 30 48 52
However, these approaches heavily depend computationally expensive per-scene optimization. To address this limitation, several new neural implicit representation methods emphasizing on generalization has emerged. These methods 6
aim to learn representation of the radiance field from a provided set of images of novel scenes. This enables the synthesis of novel views without requiring per-scene optimization. PixeINeRF (5T) and IBRNet 45) utilize volume rendering techniques to generate novel view images by employing warped features from nearby reference images. In addition to novel synthesis, MonoNeerallFusion [56] introduces a novel neural implicit scene representation with volume rendering for high-fidelity generalizable 3D scene reconstruction from monocular videos. Inspired by these generalizable methods, we takes in multi-view image feature as the input instead of the position information to enhance the generalizability of our model. Semantic knowledge obtained from 2D vision is then aggregated to infer 3D semantics by the proposed soft voting scheme.
45
49
51
5
3 Our GNeSF: Generalizable Neural Semantic Fields
We propose GNeSF to achieve generalizable semantic segmentation on neural implicit representation. As illustrated in Fig. [ our GNeSF consists of three main components: 1) Feature extraction and semantic prediction from multiple source views; 2) Geometry and semantics prediction for sampled 3D points; 3) Semantic map prediction with volume rendering.
3

---

Fastare map (hie
Query's
MVS
30
Feature Volume
Encoding Volume
Querled 30
Features MXY)
Geometry 40A)
Rey
Query X0
Weight w(O
Rendered Samantic Str)
(X)
point
Querled Faatures Ryo))
Concafemate
Voting
Nearby
T
veego
Taiget
Imuge
30 Semantica
Image
Figure 1: The GÑeSF framework comprises three key components: 1) Extraction of features and semantic prediction from multiple source views, shown the blue dashed box; 2) Prediction of geometry and semantics for sampled 3D points, shown in the yellow dashed box; 3) Prediction of semantic maps using volume rendering. shown in the red dashed box. Here, "MVS" means using Multi-View Stereo to build the feature volume.
3.1 Source- View Image Features and Semantics Map
Inspired by existing generalizable neural implicit representations 6 45 our framework leverages image features from source-view images for generalization unseen scenes during inference. We extract an image feature map F; E H for each source image I; E [0, 1), [2xW;x3 with U-Net-based convolutional neural network, where H; W; and d denotes the dimensions of the image and the feature maps, respectively. Concurrently, we use pre-trained mask2former (10) to predict the semantic maps S, E 1.,xxxixx of each source image 1,, where represents the number of categories.
3.2
3D Point Geometry and Semantics Predictions
Using the source-view image features and semantics maps, we predict the geometry (x) and semantics s(x) for each sampled 3D point x E R We first build a geometry encoding volume M based on the image feature map F, and then predict the geometry (xx) using MLP. Concurrently, we design soft voting scheme (shown in Fig. 2a) based on the 2D semantic observations S, R Wixe to predict the semanties sxx).
3.2.1 Geometry Encoding Volume
To construct the geometry encoding volume M. we project each vertex k E R of the volume to the image feature maps F, and obtain its image features F2(m, (k,)))) by interpolation, where n. (k) denotes the projected pixel location of k on the feature map F,. For simplicity, we abbreviate F, (k,,k) as F, (k). We then compute the mean f Idd and variance v Rd of the projected image features (k) from the N source views to capture the global information. The mean and variance are concatenated to build feature volume B. Intuitively, the mean image feature provides cues for the appearance information, and the variance for geometry reasoning. Finally, we use 3D neural network H, to aggregate the feature volume B to obtain the geometry encoding volume M. Formally, the whole process be expressed by:
(4k) = Mean (K) v(k) (Fr((F,(k)) B(k) [p(k), vk)), M HH,
(1a)
(1b)
where [) represents feature concatenation, and Mean(-) and Var(.) are the averaging and variance operations. For sampled 3D point x, we use an MLP network fo predict the geometry (xx) based on the interpolated features of the geometry encoding volume M(x), i.e.:
o (x) fM(())).
(2)

---

SDF
Object
T m0
0.3
X3
Ray
0-ll.
Berark
(a) Soft Voting Scheme
(b) Visibility Module
Figure 2: (a) The soft voting scheme aggregates the 2D semantic information from different views to infer semantics each 3D point. (b) The visibility module detects and climinates occluded views for each spatial point. For example, X1 is visible in the 2D semantic observation because the SDF values along the ray from the source view to the target point transits from positive to negative. On the other hand, X2 and X3 occluded in the 2D semantic observation because the value changes from positive to negative and then back to positive.
Note that geometry (x). e.g., volume density or Signed Distance Function (SDF) is represented differently in different neural implicit representations. More details are provided in Sec. 4
3.2.2 Semantics Soft Voting
Given a query point location x E Rd along a ray r E R3, we project x onto N nearby source views using the respective camera parameters. Image features (F, maps {x) E [0, are then sampled the projected pixel locations over the N source views through bilinear interpolation on the image feature F; and semantic maps S, respectively. We propose soft voting scheme to predict the semanties s(x) of a sampled 3D point x based on
Idd and semantics
the 2D projected image features (x) and semantics maps (Sx(x) JN Specifically, the 2D projected image features (F) are used to predict voting weights {w: }
i== for the corresponding
semantics maps {S:())) N in the source views.
We first calculate the mean and variance of the image features (F, J= from the N source views to capture the global consistency information. Each image feature F; (x is concatenated with the mean and variance together, and then fed into a tiny MLP network to generate a new feature (x). Subsequently, we feed the new feature (x).. the displacement Ad,
o, between the 3D point x and the camera origin o of the corresponding source view, and the trilinearly interpolated volume encoding feature M(x) into an MI.P network fo to generate voting weights (x).. In contrast to view-dependent color, we use only displacement for semantics since remains consistent regardless of the viewing direction.
Furthermore, we encode the displacement information Ad; since nearby views should contribute more compared to the distant ones. The voting weight (x()) is predicted as:
o, for voting weights prediction
(x) (x), M(x), Ad,).
The final semantics of the 3D point is obtained through a soft voting mechanism, given by:
s(x) Σ
S,(x) exp (vx(x)) Σ
exp (1)(x))
(4)
j=1
where exp(.) represents the exponential function.
Remarks. An alternative approach would be to directly regress s(x) instead of predicting the voting weights. Compared this alternative approach, predicting the voting weights makes it easier for network to leverage the source-view semantics maps which are easier to obtain without direct 3D supervision. Our conjecture verified experimentally by the degraded performance when we directly regress the semantics.
5

---

3.3 Semantics Rendering and Training
Rendering. The method presented in the previous section enables the computation of semantics s(x) and geometry o(x), e.g., volume density or SDF in the continuous 3D space. The volume rendering of the semantics S(r) along ray traversing the scene is approximated by the weighted sum of the semantics from the M samples along the ray, i.e.:
M
S(r)
= Σ ((x)) s(Xm);
(5)
m==
where the samples from to M are sorted in ascending order based on their depth values. Xm represents the query point of the m-th sample along the ray r. s(xm) and w(xx) represent the semantics and the corresponding weight of the query sample Xm, respectively. The computation of the weight w(Xm) depends on the task of view synthesis or 3D segmentation (c.f. See. for details).
Training objective. To train the model, we minimize the cross- coss-entropy error between the rendered semantics Sr)) and the corresponding ground truth pixel semantics S(r).:
L.
Σ S(r) log S(r),
R
PER
where R represents the set of rays included in each training batch.
4 Semantic View Synthesis and 3D Segmentation
Our proposed generalizable neural semantic field can be applied to the tasks of semantic view synthesis and 3D semantic segmentation.
4.1 Semantic View Synthesis
Semantic view synthesis aims to render the semantic maps for novel views. A naive approach for synthesizing novel semantic views first uses a generalizable NeRF to generate images of the novel views and then followed by applying 2D semantic segmentation to these synthesized images using separate 2D semantic segmentor. In contrast to this two-stage method, our approach is a one-step synthesis that is more efficient and streamlined. In semantic view synthesis,
the predicted geometry a (Xm)) from Eq. [] as the volume density. We then follow NeRF (30 to compute the weight w(xm) used in Eq. by:
m-
(xx)) = Txxx) (1 exp(-o(xx))), where T(xm) = exp(- a (xm)) j=1
Σ
(7)
the accumulated transmittance along the ray. Once the weights w(X.) are computed, we proceed to construct and train generalizable neural semantic field based on the approach proposed in Sec. 3.3
for rendering semantic views.
4.2
3D Semantic Segmentation
The goal of 3D semantic segmentation is to predict the semantic of each sample point in the space. Consequently, the rendering of the semantic map is dependent high-quality surfaces. To this end, we predict the geometry o (Xm) as signed distance field (SDF) based on Eq. Following (1, we then compute the weights w( (mm) by:
(x(xm)) tr
(- [xxx) tr
w (mm
Sigmoid
Sigmoid
8
where ir represents the truncation distance. We set the weights of samples beyond the first truncation region to zero to account for possible multiple intersections. Subsequently, the normalized weight (xx) used in Eq. is calculated as:
wr(xm) Ex= w((x))
w/ (xxm)
(9)

---

In this scheme, the highest integration weight is assigned to surface points, while the weights for the points that are farther away are reduced.
Visibilityy module. While most nearby views offer valuable semantic observations for 3D point x, there exists some views that contribute negatively due to occlusion. The most straightforward approach to address occlusion is to render the depth map in the same manner color semantics. However, this method incurs additional computational cost. To address occlusion efficiently, we leverage the property of signed distance function (SDF): the SDF value is positive in front of the surface and negative behind This implies that if a source view can observe a target 3D point, the SDF values of the points along the ray transit from positive to negative at the origin of the source view to the target point (c.f. Xj in Fig 2b). However, the presence of occlusion, the value changes from positive negative and then back to positive (c.f. X2 and x3 in Fig 2b). We use this property to mask out occluded views and mitigate the influence of occlusion.
Experiments
We evaluate our method the tasks of semantic view synthesis in Sec. 5.1.1 and 3D semantic segmentation in Sec. 5.1.2 Additionally, we validate the effectiveness of the proposed modules in
Sec. 5.2
Dataset and Metrics. We perform the experiments on two indoor datasets: ScanNet (V2) [3 and Replica 38 The ScanNet dataset contains 1, 613 indoor scenes with ground-truth camera poses, surface reconstructions, and semantic segmentation labels. We follow [31] and 40 on the three training/valiiaationnnes splits commonly used in previous works. The 2D semantic segmentor Fask2Former is pre-trained on the train split of ScanNet without any additional amendment. Further- more, follow the 20-class semantic segmentation task defined in the original ScanNet benchmark. In contrast to NeRF-based methods [53] 561 which only use the training subset of ScanNet to train and test their methods, we train and test our method on the complete dataset. Replica consists of 16 high-quality 3D scenes of apartments, rooms, and offices. We use the same 8 scenes as in [55]. While Replica originally features 100 classes, we rearranged them into 19 frequently observed semantic classes. We use the model trained on ScanNet to perform the validation on Replica. For the evaluation metrics, we use the mean of class-wing wise intersection over union (mloU) and the average pixel accuracy (mAcc). For 3D semantic segmentation, we follow the evaluation procedure of Atlas 31
Table 1: Quantitative comparison on semantic view synthesis from the validation set of ScanNet. 10urs-100' means the model trained on 100 scenes and 'Ours-1000' trained on full training set of ScanNet. Semantic-Ray is also trained full training set of ScanNet. 'Predict Directly' represents the method that incorporates a semantic head into the generalizable NeRF framework.
Novel View
Novel Scene
Method
mloU (%)
mAcc (%)
mloU (%)
mAcc (%)
Semantic-NeRu
55
96.8
98.2
Per-scene
DM-NeRF
3
93.5
96.1
optimized
PNF 23
94.1
97.2
Fask2Former
110
80.9
90.7
64.6
77.9
Predict Directly
50.2
62.3
39.1
54.2
Gneralizable
Semantic-Ray
24
56.0
Ours-100
93.3
96.3
47
60.8
Ours-1000
87.8
93.3
71.6
82.3
5.1
Comparisons to Baselines
5.1.1
Semantic View Synthesis
We compare with state-of-the-arr per-scene optimized neural semantic field methods: Semantic-NeRR (55, Panoptic Neural Fields (PNF) (3 and DM-NeRF [3] For each per-scene optimized method,
train NeRF model each scene from the train set of ScanNet, and then compute the 2D mloU
7

---

across these scenes. Note that although PNF and DM-NeRF aim to predict panoptic segmentation, they can still predict semantic segmentation. We train our method for different number of scenes, including 100 scenes (denoted as Ours-100) and full training set (denoted as Ours- 1000), to show the capacity of our model over different scenarios. For generalizable methods, we compare with Fask2Former, Sematic-Ray [24] and the method adding a semantic head to a generalizable NeRF (denoted as Predict Directly) the same training and testing setting. For Mask2Former, IBRNet 45 is first used to render color images novel views, and then Mask2Former is used to predict semantic results on rendered images. For Predict Directly, we use IBRNet as its generalizable NeRF framework.
As shown in Tab. ! we outperform the baselines across ScanNet on semantic segmentation tasks. Compared with per-scene optimized methods, our method can obtain comparable performance (93.3% v.s. 93.5% 2D mloU) on novel views. Moreover, only our methods can be generalized to novel scenes (47% 2D mloU). Although Semantic-NeRF performs better than our method (++.5%%) it requires per-scene optimization and thus not generalizable to unseen scenes, Comparing with other generalizable methods, our method obtain the state-of-the-art performance. For example, our method significantly improves over Mask2Former by 7%, which shows our method better and more concise. Furthermore, our improves significantly (71.6% v.s. 56%) compared with Sematic-Ray. In addition, our method has improved tremendously (71.6% v.s. 39.1%) over the naive approach that incorporates semantic head into the generalizable NeRF framework. This validates the effectiveness of our soft voting scheme.
Table 2: Quantitative comparison 3D semantic segmentation from the val and test set of ScanNet. Note that 'Geometry' means using 3D geometry as input and 'Sup.' represents supervision.
Method
Val Split (mloU)
Test Split (mloU) Geometry
Atlas
3
36.8
34
3D Sup.
NeuralRecon
40
37.5
35.1
Joint Recon-Segmen
29
51.5
X
PointNet++
33
53.5
33.9
PointConv
47
61
55.6
S4R
20
36.9
ScanComplete 15
29.5
2D Sup.
Ours
58.8
55.1
Ours with Mesh
60.4
55.8
5.1.2 3D semantic segmentation
We evaluate 3D semantic segmentation the val and test set of ScanNet. We compare our method with approaches trained on 3D semantic supervision, iee. Atlas (31), NeuralRecon [40]. Joint Recon-Segment 22), PointConv 47.. and PointNet++ 33 and approaches trained on 2D semantic
supervision, i.e., S4R 20)) and ScanComplete [15.. The input of these methods are different: PointConv 47). PointNet++ (33). S4R (20). and ScanComplete [15] use 3D geometry, c.g., point cloud, 3D mesh etc, as input; Others, i.e., Atlasm NeuralRecon, and Joint Recon- SeeonnS [2 use RGB images as input to predict 3D surfaces and use an extra semantic head predict semanties. In Tab. we use 'Geometry' to denote 3D geometry as input and 'Sup.' to represent supervision.
As shown in Tab. D. our proposed method achieves comparable performance to the methods with 3D semantic supervision, and significantly outperforms all methods with 2D semantic supervision. For example, our approach performs significantly better than all other methods using RGB images as
Table 3: Quantitative comparison Replica.
Method
Atlas
NeuralRecon
Ours
3D mloU
25.4
26.2
43.8
8

---

NeuralRecon
Ours
wall
cablnet
refrigerator
desk
bookshetf
picture
countar
curtain
bed
shewer curtain
bollet
sink
bathtub
otherfarniture
ipnored
Qualitative 3D semantic segmentation results on ScanNet. As we can see from the figure, our method is better than NeuralRecon, especially the 'bed' class. In detail, NeuralRecon mistakenly classifies the bed as a table (highlight in black box).
Figure 3:
input. The highest mloU of the previously best method that predicts 3D surface on the ScanNet test set is 51.5% 29), which significantly less than our results of 55. 1%. More importantly, they need 3D semantic supervision while we only require 2D semantic supervision. Our approach still performs better than some of the methods (58.8% v.s. 53.5% (33) with 3D supervision and 3D geometry input. Our method is comparable to PointConv which uses 3D Geometry as input (60.4% v.s. 61%) when we replace SDF prediction with ground truth. Similar to our method, S4R leverages differentiable rendering training supervision. However, S4R incorporates semantic head into the existing NeRF framework and uses 3D geometry as input. Our approach outperforming S4R (36.9% 58.8% proves the effectiveness of proposed soft voting scheme. We also show qualitative comparison with NeuralRecon in Fig. We can see that NeuralRecon wrongly classifies the bed table (highlight in black box), while approach makes the correct prediction. Tab. shows the performance of our method compared with Altas [31] and NeuralRecon [40] that also use color images input when trained ScanNet and test Replica. We that method shows much stronger ability (43.8% v.s. 75.4%%26.2% for cross-dataset generalization.
5.2 Ablation Studies
As shown in Tab 4 we do ablation study investigate the contribution of each component proposed our method on ScanNet. We start with a baseline method predicting the semantics directly of each 3D query point. Next, we use only semantic logits to vote for semantics of each 3D query point. We then performed a series of tests where we include each component our framework one-by-one and measured the impact a probability distribution instead of semantic logits, and the fourth shows the impact adding the visibility module. We find that each component improves the 3D segmentation IoU performance.
the performance. The third row shows the impact of using
Table 4: Ablations of our design choices 3D semantic segmentation.
Predict Directly Logit
Prob.
Visibility Module
mloU
mAcc
37.5 (+0)
47.0 (+0)
Xx1
x
X
57.3 (+19.8)
70.9 (+23.9)
X
/
X
57.9 (+0.6)
71.2 (+0.3)
X
X
V
/
58.8 (+0.9)
72.1 (+0.9)

---

6 Conclusion
In this paper, we introduce a generalizable 3D segmentation framework based on implicit repre- sentation. Specifically, we propose novel soft voting mechanism on 2D semantic information from multiple viewpoints to predict the semantics for each 3D point. Instead of using positional information, our framework utilizes multi-view image features input and thus avoids the need to memorize scene-specific geometric and semantic details. Additionally, our framework encodes view differences to give higher importance nearby views compared to distant ones when predicting the voting scores. We also design a visibility module to identify and discard detrimental information from occluded views. By incorporating view difference encoding and visibility prediction, framework achieves more effective aggregation of the 2D semantic information. Experimental results demonstrate that our approach performs comparably to scene-specific methods. More importantly, our approach even surpasses existing strong supervision- -aased approaches with just 2D annotations.
Acknowledgement
This research work is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L.60021).
References
[1] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6290-6301, June 2022.
[2] Aude Billard and Danica Kragic. Trends and challenges in robot manipulation. Science, 364(6446)ceu88114 2019.
[3] Wang Bing, Lu Chen, and Bo Yang. Dm-nerf: 3d scene geometry decomposition and manipula- tion from 2d images. arXiv preprini arXiv:2208.07227) 2022.
[4] Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver. Visual navigation for mobile robots: A survey, Journal of intelligent and robotic systems, 53:263-296, 2008
[5] Ang Cao, Chris Rockwell, and Justin Johnson. Fwd: Real-time novel view synthesis with forward warping and depth. CVPR, 2022.
[6] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124- 14133, 2021.
[7] Liang-Chiae Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 4044) 848, 2017.
[8] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint 1rXiv:1706.05587 2017.
[9] Shenchang Eric Chen and Lance Williams. View interpolation for image synthesis. In Proceed- ings of the 20th annual conference Computer graphics and interactive techniques, pages 279-288, 1993.
[10] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-ate basked-attention mask transformer for universal image segmentation. 2022.
[11] Christopher Choy, Jun Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings the Vision and Pattern Recognition, pages 3075-3084 2019.
the IEEE Conference on Computer
10

---

[12] Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features, In Proceedings of the IEEE/CVF international conference on computer vision, pages 2019.
[13] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotater 3d reconstructions of indoor scenes. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 5828-5833 2017.
[14] Angela Dai and Matthias Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 452-468, 2018.
[15] Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias Nießner. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2018.
[16] Zhiwen Fan, Peihao Wang. Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. arXiv preprint arXiv:2209.08776 2022.
[17] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, An- dreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In International Conference on 3D Vision (3DV), 2022.
[18) Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshenn: A network with an edge. In Proceedings of the IEEE International Conference on Computer Vision, pages 9365-9374, 2019.
[19) Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongbo Fu, and Chiew-lan Tai. Jsenet: Joint semantic segmentation and edge detection network for 3d point clouds. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XX 16, pages 222-239. Springer, 2020.
[20] Junwen Huang, Alexey Artemorv, Yujin Chen, Shuaifeng Zhi, Kai Xu, and Matthias Niess- ner. S4r: self-super semantic scene reconstruction from rgb-d scans. arXiv preprint arXiv:2302.036440 2023.
[21] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems, volume 35, 2022.
[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in neural information processing systems, volume 25, pages (097-1105, 2012.
[23] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12871-12881 2022.
[24] Fangfu Liu, Chubin Zhang, Yu Zheng, and Yueqi Duan. Semantic ray: Learning a generalizable semantic field with cross cross-repro cooss-reroojcction attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[25] Xinhang Liu, Jiaben Chen, Huai Yu, Yu-Wing Tai, and Chi-Keung Tang. Unsupervised multi- view object segmentation using radiance field propagation. Advances in Neural Information Processing Systems, 35:17730-17743, 2022.
[26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
11

---

[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[28] Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic con- volutional neural networks riemannian manifolds. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 37-45, 2015.
[29] Davide Menini, Suryansh Kumar, Martin Oswald, Erik Sandström, Cristian Sminchisescu, and Luc Van Gool. A real-time online learning framework for joint 3d reconstruction and semantic segmentation of indoor scenes. IEEE Robotics and Automation Letters, 7(2):1332-1339, 2021.
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European Conference Computer Vision, pages 405-421. Springer, 2020.
[31] Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-end 3d scene reconstruction from posed images. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow. UK, August 23-28, 2020, Proceedings, Part VII 16, pages 414 431. Springer. 2020.
[32] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 652-660, 2017.
[33] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning point sets in metric space. Advances in neural information processing systems, 30, 2017.
[34] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
[35] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representa- tions at high resolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3577-3586, 2017.
[36] Olaf Ronneberger, Philipp biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
Fischer, and Thomas Brox. U-net: Convolutional networks for
[37] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529-10538, 2020.
[38] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, al. The replica dataset: digital replica of indoor spaces. arXiv preprint arXiv: 100xv:9906.000 19066057977 2019.
[39] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: semantic segmentation. In Proceedings of the IEEF/CVF international conference on computer vision, pages 7262-7272, 2021.
Transformer for
[40] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real- time coherent 3d reconstruction from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15598-156077 2021.
[41] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. 2021.
12

---

[42] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE International Conference Computer Vision, pages 4489 4497, 2015.
[43] Vadim Tschemezki,, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3D distillation of self-supervised 2D image representations. In Proceedings of the International Conference 3D Vision (3DV), 2022.
[44] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes, 2021.
[45] Qianqian Wang, Zhicheng Wang. Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibmet: Learning multi-view image-based based rendering. In CVPR, 2021.
[46] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7794-7803, 2018
[47] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings the IEEE/CVE Conference computer vision and pattern recognition, pages 9621-9630, 9630, 2019.
[48] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Citynerf: Building nerf at city scale. arXiv preprint 1rX152211..05504, 2021.
[49] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. arXiv preprint arXiv:2201.086 208845, 2022.
[50] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning dojjectcommositionns neural radiance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13779-13788, 2021.
[51] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one few images. In CVPR, 2021.
[52] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[53] Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radiance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5449-5458, 2022.
[54] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2881-2890, 2017.
[55] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. 2021.
[56) Zi-Xin Zou, Shi-Sheng Huang, Yan-Pei Cao, Tai-Jiang Mu, Ying Shan, and Hongbo Fu. Mononeuralfusion: Online monocular neural 3d reconstruction with geometric priors. arXiv preprint arXiv:2209. 15153, 2022.
13


Reviews:
[{"review_id": "5Qa07JJjWB", "reviewer": "Reviewer_xsYz", "summary": "This paper proposes a generalizable 3D segmentation framework. The novel components include (a) a soft voting mechanism that incorporates image features, view difference aggregate the 2D semantic information for 2D views to 3D, (b) a visibility module to avoid being affected by occluded views. With only 2D supervision, this method allows for semantic maps synthesis and 3D semantic segmentation for novel 16 scenes. The method is also strongly supported by experiments.", "questions": "Tab. 1 shows that ours-1000 achieves a significant improvement over ours-100 on novel scenes. Have the authors tried on even larger set of training scenes? Could the authors provide some insights on when the marginal gain on expanding the training set will be negligible? How many scenes are Mask2Former and Predict Directly trained on?", "limitations": "The authors have not adequately addressed the limitations or potential negative societal impact. The authors could show some failures cases that indicate some common patterns where the proposed method wouldl fail on.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The ability to generalize is very important in many tasks for efficiency. This paper proposes a generalizable neural semantic field, which shows significant improvements over prior work on unseen scenes. \n\n2. Compared to prior work that uses 3D supervision, this method only needs 2D semantic supervision during training and can be very helpful for 3D segmentation tasks whose annotations are very costly to obtain.\n\n3. Despite being a bit complex, the proposed model includes a few novel and thoughtful components, which may be in helpful to inspire the community.", "weaknesses": "1. Not enough intuition is given to explain the qualitative results. Fig. 3 shows that NeuralRecon predicts worse than the proposed method on bed. However, the authors didn't explain why this can happen (which design choice has potentially helped with this)? In the results shown in supplementary Fig. 2, it seems the proposed method has many dotted predictions (eg, red dots on the walls in the second row), what does this mean? Does it mean the proposed method may have some problem predicting the doors on the wall?\n\n2. The clarity of the figures can be improved. For example, \"MVS\" in Fig. 1 is never explained in the text. The names of the features and neural networks in Fig. 1 sometimes are not exactly the same as the names used in the text, making it a bit hard for the readers to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Respose to Reviewer xsYz (R#1)\n\n**Q1**: Not enough intuition is given to explain the qualitative results. Fig. 3 shows that NeuralRecon predicts worse than the proposed method on bed. However, the authors didn't explain why this can happen (which design choice has potentially helped with this)? \n\n**A**: NeuralRecon does not perform well on ‘bed’ due to the lack of training data for this category as it is on the long-tail end of the scannet dataset. In contrast, our proposed method introduces a soft voting scheme to compute the semantics based on the large pretrained 2D segmentation model. Since the 2D segmentation model is pretrained on large-scale 2D dataset  (e.g. ImageNet21K) that includes abundant data for beds, our proposed method can therefore work well on the ‘bed’ category.\n\n**Q2**: The proposed method has a few dotted predictions, what does this mean?\n\n**A**: The dotted predictions are caused by the inconsistency between nearby views. In most cases, this inconsistency can be remedied by learning proper weights in our soft voting method. However, the dotted phenomenon will appear in some extreme cases where the 2D predictions of most views are inconsistent with each other.\n\n**Q3**: The clarity of the figures can be improved. For example, \"MVS\" in Fig. 1 is never explained in the text. The names of the features and neural networks in Fig. 1 sometimes are not exactly the same as the names used in the text, making it a bit hard for the readers to understand.\n\n**A**: Thanks for your suggestions. “MVS” means using Multi-View Stereo to build the cost volume. $x^{3d}$ in Fig. 1 is $x$. We will make these clearer in the final version.\n\n**Q4**: Tab. 1 shows that ours-1000 achieves a significant improvement over ours-100 on novel scenes. Have the authors tried on even larger set of training scenes? Could the authors provide some insights on when the marginal gain on expanding the training set will be negligible? \n\n**A**: We did not try on training scenes larger than ScanNet. This is because: 1) We follow the previous works [49,37,51] to use the ScanNet in our experiments for fair comparison. 2) ScanNet is sufficiently large with a training set of 1,201 scenes, a validation set with 312 scenes and a test set with 100 scenes. Nonetheless, to analyze the effect of the number of training scenes, we conduct an experimental comparison by setting the number of training scenes to 100, 200, 500, 800, 1000 and 1201 (full training set), respectively. We can see from the table below that the performance is improved when more scenes are used. However, the improvement is marginal when the training scene is increased from 1000 to 1201.\n\n|   # Scenes  | Novel View |          | Novel Scene |          |\n|:-----------:|:----------:|:--------:|:-----------:|:--------:|\n|             |  mIoU (%)  | mAcc (%) |   mIoU (%)  | mAcc (%) |\n|     100     |    93.3    |   96.3   |     47.0    |   60.8   |\n|     200     |    92.2    |   95.7   |     58.2    |   69.8   |\n|     500     |    91.0    |   94.9   |     65.8    |   76.6   |\n|     800     |    89.7    |   94.3   |     68.9    |   79.4   |\n|     1000    |    88.6    |   93.7   |     71.2    |   81.5   |\n| 1201 (full) |    87.8    |   93.3   |     71.6    |   82.3   |\n\n\n\n**Q5**: How many scenes are Mask2Former and Predict Directly trained on?\n\n**A**: Both Mask2Former and Predict Directly are trained on the 1201 training scenes of ScanNet, which is the same as our method. The reviewer can also refer to line 245 of the main paper for further description.\n\n**Q6**: The authors have not adequately addressed the limitations or potential negative societal impact.\n\n**A**: Although our proposed method benefits from 2D segmentation in most cases, our proposed method cannot work well in extreme cases when 2D segmentation cannot segment objects correctly. Furthermore, the task of the proposed method tries to understand the real world, which could infringe on individuals' privacy rights and raise concerns about surveillance.\n\n**Q7**: The authors could show some failure cases that indicate some common patterns where the proposed method would fail on.\n\n**A**: As discussed in Q2, there are a few dots when the predictions between nearby views from a pre-trained 2D semantic segmentation are inconsistent. Although our method can predict correct semantics, there are still a few wrong parts which manifest themselves in the form of spots as shown in Fig. 1 of the PDF file in the rebuttal. For example, the red dots in the third row on the chair, and the yellow dots in the fourth row on the sofa.\n"}, {"review_id": "jOYsWHSV5T", "reviewer": "Reviewer_kUSs", "summary": "This paper proposed GNeSF, a generalizable neural semantic fields by incorporating contextual and semantic prior information via a U-shaped CNN and a pre-trained segmentor. Combing multi-view fused image-level features as well as soft weighting using multi-view softmax probabilities, the overall pipeline could be trained on diverse scenes and then generalize to unseen scenes with promising performance.", "questions": "Please see the comments in the weaknesses section above. My main concern comes from missing comparison to closely related work like Semantic-Ray  and lack explanations of several key components, which makes the technical contribution not clear in my point of view.", "limitations": "There is no clear discussion of limitations and potential negative societal impact in the main paper and supplement.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes a valid extension to enable a generalizable NeSF model by incorporating deep image features as well as semantic priors of multiple RGB images.\n- Promising performance on ScanNet and Replica datasets on 2D/3D shows the effectiveness of the methods. Experiments are relative adequate.", "weaknesses": "- Contributions and novelties.\n  -  As far as I know, Semantic-Ray [1] from CVPR2023 has already made attempt on learning a generalizable semantic NeRF model via attention, which shares similar spirits to GNeSF. What is the main difference?\nMy main concerns comes from the lack of necessary comparison and discussions to related baselines such as Semantic-Ray [1] and other generalizable NeRF (NeuRay, MVSNeRF) with a semantic head in order to make the distinction clear.\n\n  - Similarly, the discussion of using logits vs. prob. has also been somewhat investigated in PanopticLifting [2] from CVPR2023, thus further weakens the contributions.\n\n- As to numerical experiments, several necessary information is missing in both main paper and supplement and is a bit hard for readers to really judge the strict fairness of comparison.\n   - Supervision of using predicted multi-channel prob tensors vs. hard-labels for baselines. If I understand correctly, multi-channel softmax probabilities are predicted per view to serve for soft voting. For baseline methods, are these multi-channel prob tensors also used for training like [2] or only hard discrete labels are used for training baselines which lost categorical information across classes? \n  - How many nearby views are used for building the cost volumes and how does the number of views affect the final performance?\n  - I also wonder what is the specific dimension/resolution of the built cost volume? As to cover the whole scene, I assume the cost volume owns a large spatial dimension and may make the training costly? Its dimension will affect the represented level of details of deep features.\n\n- Ablations\n  - It seems that the mentioned design choices contributes quite marginally to the final performance as shown in Table4\n  - In Section 3.2, authors discusses the motivation why soft voting is used instead of directly regressing semantics. It would be good if related results could be shown in ablations (like Table4).\n\n\nOther minor comments which need further clarify:\n - It would be good to show the training expenses of GNeSF.\n- What is exactly the adopted generalizable NeRF in the experiments (i.e., Predict Directly)?\n- How to evaluate 3D segmentation as there are discrepancy between reconstructed mesh and GT mesh. Is the same procedure in Atlas used?\n\n\n\n\n[1]Semantic Ray: Learning a Generalizable Semantic Field with Cross-Reprojection Attention CVPR 2023\n\n[2]Panoptic Lifting for 3D Scene Understanding with Neural Fields CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Respose to Reviewer kUSs (R#2)\n\n**Q1**: What is the main difference with Semantic-Ray [a].\n\n**A**: Although Semantic-Ray [a] and our method both attempt to learn a generalizable semantic NeRF model, the approach taken by the two methods to solve this problem differs significantly. Semantic-Ray proposes a new attention-based feature fusion mechanism for each 3D point and then predicts semantics directly. Specifically, Semantic-Ray fuses features of a 3D point by intra-view radial attention and cross-view sparse attention. Instead of focusing on feature fusion and predicting semantics directly, which is the core contribution of Semantic-Ray, our method focuses on a novel soft voting mechanism to aggregate the 2D semantic information from different views for each 3D point. The advantage of our method is utilizing the SOTA 2D segmentation model to vote for 3D semantics.\n\n**Q2**: Compare with Semantic-Ray and other generalizable NeRF.\n\n**A**: To compare with Semantic-Ray, NeuRay, and MVSNeRF fairly, we train our model with the training scenes from the setting Semantic-Ray and test on the same 10 scenes with Semantic-Ray, shown in the first row of the Tab. 1 of the PDF file in the rebuttal. The results of Semantic-Ray, NeuRay, and MVSNeRF are from the paper of Semantic-Ray. To show the advantage of utilizing abundant 2D data and a large 2D pre-trained model, we use the same setting as our method to train Semantic-Ray on 1201 scenes and test on 312 scenes, shown in the second row of the Tab. 1 of the PDF file in the rebuttal. We can see from the table our method obtains marginal improvement (58.47 v.s. 57.15) when training on 60 scenes compared with S-Ray, but improves a lot (71.62 v.s. 56) when training on 1201 scenes.\n\n\n**Q3**: The discussion of using logits vs. prob. has also been investigated in PanopticLifting [b], weakening the contributions.\n\n**A**: Our core contribution is the proposed novel soft voting mechanism to aggregate the 2D semantic information from different views for each 3D point. We did not claim the use of prob. as our contribution in the main paper. Using prob. is more like a choice of design. Moreover, prob. contributes marginally to final performance as shown in Tab. 4. This shows that our  proposed softing voting method does not critically rely on the prob.\n\n**Q4**: For baseline methods, are multi-channel prob tensors used for training like PanopticLifting [b]?\n\n**A**: For fair comparison, we follow [51, 10] in using hard discrete labels from the ground truth to train all baseline methods and our method. Nonetheless, PanopticLifting [b] utilizes 2D semantic segmentation pretrained on other dataset to generate pseudo labels that are represented as multi-channel prob tensors to optimize the model. This is because PanopticLifting is a per-scene optimized NeSF that cannot infer novel scenes directly without training. The generated pseudo labels are therefore used as surrogates to the ground truth labels that are absent on the novel scenes.\n\n**Q5**: What is the dimension of the cost volume? Does the training cost a lot? The dimension will affect the represented level of details of features.\n\n**A**: We follow Neuralrecon [37] and Mononeuralfusion [52] for constructing the cost volume. Specifically, we first build local cost volumes with resolution $96\\times96\\times96$, which are then fused into a global one. The local cost volume is processed with 3D sparse convolution and the global cost volume is saved as sparse tensors. Consequently, the training process does not cost much and is similar to [37, 52]. The dimension indeed affects the represented level of details of features because more voxels can be used to represent 3D space, and each voxel can represent smaller structures and details when the dimension is larger.\n\n**Q6**: How many nearby views are used for building cost volumes. How does the number of views affect the final performance?\n\n**A**: $10$ nearby views are used for building the local cost volumes. The effect of the number of views is ablated in the Tab. 2 of the PDF file in the rebuttal. We can see that better performance can be obtained with more views. However, the improvement becomes marginal when the number of views is more than 10. Based on this observation,  we set the number of nearby views to 10.\n\n**Q7**: The mentioned design choices contribute marginally to the final performance as shown in Tab. 4. Tab. 4 should include the result of Predict Directly.\n\n**A**: Tab. 4 does not include our core contribution of soft voting. Instead, we compare soft voting with NeuralRecon that uses directly regressing semantics (Predict Directly) in Tab. 2. We now combine Tab. 2 and 4 in the Tab. 3 of the PDF file in the rebuttal, where we can observe that our proposed soft voting method improves 19.8% mIoU over directly regressing semantics.\n\n\n**Q8**: About the training expenses of GNeSF.\n\n**A**: For Semantic View Synthesis, we train GNeSF about 30 hours on a single RTX3090 GPU with 24GB memory. For 3D semantic segmentation, GNeSF require 34 hours for features learning as well as geometry prediction and 7 hours for semantics prediction with 4 RTX3090 GPUs.\n\n**Q9**: What is the adopted generalizable NeRF as a baseline?\n\n**A**: Our adopted generalizable NeRF baseline is IBRNet [42] with an additional semantic head for semantics.\n\n**Q10**: How to evaluate 3D segmentation?\n\n**A**: We follow the evaluation procedure of Atlas as mentioned in Line 253. The semantic labels from the predicted mesh are transferred onto the ground truth mesh using nearest neighbor lookup on the vertices, and then the standard IoU metric can be used.\n\n**Q11**: Discuss limitations and societal impact.\n\n**A**: The answers can be found in R#1Q6.\n\n[a] Fangfu Liu, etc. Semantic Ray: Learning a Generalizable Semantic Field with Cross-Reprojection Attention. CVPR, 2023.\n[b] Yawar Siddiqui, etc. Panoptic Lifting for 3D Scene Understanding with Neural Fields. CVPR, 2023."}, {"review_id": "G80A8zHyy1", "reviewer": "Reviewer_1Moh", "summary": "This paper, titled GNeSF, focuses on tackling the challenge of generalizable semantic segmentation using NeRF representation for both novel segmentation synthesis and 3D segmentation. The authors introduce several key contributions in this regard. Firstly, they propose utilizing a cost volume to construct a comprehensive global feature volume. Secondly, a pre-trained 2D segmentor is employed to extract semantic information from the source views. Finally, a soft voting mechanism is introduced to predict the ultimate segmentation label for the target pixel. These approaches collectively enhance the overall effectiveness and robustness of the segmentation process.", "questions": "The authors primarily focus on showcasing the results of semantic segmentation in their work. However, it is equally important to assess the quality of view synthesis (color image) when rendering novel views. It would be beneficial if the authors could compare their approach with several representative generalizable NeRF methods on the Scannet dataset. I appreciate the experimental setup and would be inclined to give a higher score if the raised concerns are adequately addressed.", "limitations": "See the weakness and quations above.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors tackle a crucial problem in 3D scene understanding by extending the 2D segmentation map into the 3D space, enabling segmentation for any viewpoint.\n2. The framework design presented by the authors appears logical and well-founded. Leveraging the cost volume proves to be an effective approach for encoding comprehensive scene information, while the utilization of a pre-trained 2D model allows for accurate inference of per-view semantics.\n3. The introduced voting scheme successfully addresses the challenge of visibility when assigning labels to new pixels. Notably, the experimental results demonstrate a noteworthy improvement compared to previous methods, further validating its efficacy.", "weaknesses": "1. Lack of references: The authors utilize the cost volume as a crucial representation for encoding 3D scene information. However, they do not provide explicit details regarding the construction process, nor do they reference important relevant papers in the deep learning era [1][2]. Furthermore, the survey conducted on the \"Neural Semantic Field\" appears insufficient, as it fails to mention several per-scene segmentation methods based on NeRF representation [3][4][5].\n\n2. Missing technical details: The meaning of \"each vertex\" mentioned in line 151 is unclear. Did the authors explicitly construct scene volume grids by determining the near and far plane for each view? Additionally, the process of interpolating the semantic logits (line 170) is not adequately explained. Furthermore, it is unclear why the authors utilize the distance between each point x and its camera origin (line 178). Given that this is an important contribution of the proposed method, it is highly recommended to highlight the motivation for using \\delta_d at the beginning of this section.\n\n3. Experimental results: It is unclear why the novel view synthesis quality decreases from 93.3 to 87.8 in Table 1 when including more scenes for training. Could you clarify if the term \"novel view\" refers to evaluating the training scenes while leaving several views for evaluation?\n\n[1] Mvsnet: Depth inference for unstructured multi-view stereo\n[2] Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching\n[3] Nerf-sos: Any-view self-supervised object segmentation on complex scenes\n[4] Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering\n[5] Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n## Respose to Reviewer 1Moh (R#3)\n\n**Q1**: Lack of references and provide explicit details regarding the construction process.\n\n**A**: Thanks for your suggestions. We will add all mentioned references into the Related Work Section of the final version. For example, we will insert the following sentences into line 110: ‘NeRF-SOS [a] combines object segmentation and neural radiance fields to segment objects in complex real-world scenes using self-supervised learning. [b] proposes an object- compositional neural radiance field. RFP [c] designs an unsupervised approach to segment objects in 3D during reconstruction using only unlabeled multi-view images of a scene.’ The construction process of the cost volume is that to construct the geometry encoding volume, we project each vertex of the volume to the image feature maps and obtain its image features by interpolation. We then compute the mean and variance of the projected image features from the source views to capture the global information. The mean and variance are concatenated to build a cost volume. More description can be found from Line 151 to 157, and the similar building process of feature/cost volume can be found in [37,52,d]. We will add these in the final version. And  [d] is not cited in our submission, we will add it into our final version.\n\n**Q2**: Missing technical details: The meaning of \"each vertex\" mentioned in line 151 is unclear. Did the authors explicitly construct scene volume grids by determining the near and far plane for each view?\n\n**A**: We explicitly construct cost volume grids locally with the resolution $96\\times96\\times96$ based on some nearby views, and then fuse it into a global volume, following Neuralrecon [37] and Mononeuralfusion [52]. Each vertex of the cost volume grids is projected to the image feature maps and obtains its image features by interpolation. The features of each vertex is the mean and variance of projected image features across views. More details can be found in [37,52,d].\n\n**Q3**: Additionally, the process of interpolating the semantic logits (line 170) is not adequately explained.\n\n**A**: The process of interpolating the semantic logits is similar to the process of interpolating the image features. In detail, when applying bilinear interpolation, each target pixel is determined by taking a weighted average of the four nearest pixels from the original semantics map. These four pixels form a square region around the target pixel. The weights are computed based on the relative distances of the target pixel to the four neighboring pixels. \n\n**Q4**: Furthermore, it is unclear why the authors utilize the distance between each point x and its camera origin (line 178). Given that this is an important contribution of the proposed method, it is highly recommended to highlight the motivation for using $\\Delta d$ at the beginning of this section.\n\n**A**: Thanks for the suggestion. We will highlight the motivation at the beginning of the section. As shown in Line 182 and 183, the reason for encoding the distance information for voting weights prediction is that nearby views should contribute more compared to the distant ones. In other words, the predicted weight is inversely proportional to distance difference.\n\n**Q5**: Experimental results: It is unclear why the novel view synthesis quality decreases from 93.3 to 87.8 in Table 1 when including more scenes for training. Could you clarify if the term \"novel view\" refers to evaluating the training scenes while leaving several views for evaluation?\n\n**A**: Ours-1000 performs worse than Ours-100 (87.8 v.s. 93.3) on novel views, but much better than Ours-100 (87.8 v.s. 93.3) on novel scenes. This means that the model trained with more scenes Ours-1000 has better generalization capacity compared with ours-100. The reason is that it is easier for the model to overfit when training on 100 scenes compared to 1000 scenes. Consequently, Ours-100 performs better on novel view synthesis for training scenes, while performs worse on novel testing scenes. Please be noted that the ‘novel view’ refers to testing on the same scenes as training but different views, and ‘novel scene’ means that testing on different scenes from training. \n\n**Q6**: it is equally important to assess the quality of view synthesis (color image) when rendering novel views. It would be beneficial if the authors could compare their approach with several representative generalizable NeRF methods on the Scannet dataset.\n\n**A**: We provide the comparison on ScanNet in the table shown below. The results of other generalizable NeRF methods are from recent SOTA SurfelNeRF [e]. Our method uses the same scene split as SurfelNeRF for training and testing for fair comparison. From the table, we can see that although novel view synthesis is not our focus, our method is still comparable with most generalizable NeRF methods, and worse than the SOTA SurfelNeRF.\n\n|     Methods    | PSNR↑ | SSIM↑ | LPIPS↓ |\n|:--------------:|:-----:|:-----:|:------:|\n|   IBRNet [42]   | 21.19 | 0.786 |  0.358 |\n|  NeRFusion [49] | 22.99 | 0.838 |  0.335 |\n| PointNeRF [f] | 20.47 | 0.642 |  0.544 |\n| SurfelNeRF [e] | 23.82 | 0.845 |  0.327 |\n|      Ours      | 22.01 | 0.765 |  0.319 |\n\n\n[a] Zhiwen Fan, etc. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. ICLR, 2023.\n\n[b] Bangbang Yang, etc. Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering. ICCV, 2021.\n\n[c] Xinhang Liu, etc. Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation. NeurIPS, 2022.\n\n[d] Xiaoxiao Long, etc. SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views. ECCV, 2022.\n\n[e] Yiming Gao, etc. SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes. CVPR, 2023.\n\n[f] Qiangeng Xu, etc. Point-NeRF: Point-based Neural Radiance Fields. CVPR, 2022.\n"}, {"review_id": "mgsJnly4pp", "reviewer": "Reviewer_G1Te", "summary": "This paper introduces a generalizable 3D scene segmentation framework based on neural implicit representation. The proposed framework takes multi-view image features and semantic maps as inputs to reconstruct 3D geometric and semantic information.  The generalizability of the approach enables the synthesis of semantic maps or conducting 3D semantic segmentation for novel scenes with only 2D semantic supervision. Experimental results demonstrate comparable performance with scene-specific approaches and  outperform existing the methods using only 2D annotations.\n", "questions": "1. Can the network learn geometry information solely based on equation (6)? If so, why? Is there an intuitive way to explain this?\n2. Why does Ours-1000 perform worse than Ours-100?\n", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This method is novel, and similar approaches have not been proposed in previous papers.\n2. This task is useful and also quite innovative.\n3. This paper's experiments are comprehensive and thorough.\n\n\n", "weaknesses": "1. The article lacks clear explanations for many details, such as the detailed network structure and training strategy. This leads to poor reproducibility.\n2. The article claims that only 2D supervision is required, but it does not explain how the SDF (signed distance function) is learned. If following the training pipeline in NeuralRecon or Mononeuralfusion, then this claim of \"only needing 2D supervision\" is incorrect.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Respose to Reviewer G1Te (R#4)\n\n**Q1**: The article lacks clear explanations for many details, such as the detailed network structure and training strategy. This leads to poor reproducibility.\n\n**A**: We will release our source code and trained models after paper acceptance. We have provided some implementation details in supplementary material but will give further explanation here. For the network structure of 3D semantic segmentation shown in Fig. 1, the 2D CNN is MnasNet [a], the 3D encoder is SPVNAS [b], the FC used to predict geometry is one fully connected linear layer, and the FC used to predict weights is three fully connected linear layers. More details about the 2D network and the 3D encoder can be found in NeuralRecon [37]. For semantic field training strategy, we randomly sample a scene and 512 rays as the training batch for each iteration. Furthermore, following NeRF [27], we perform hierarchical volume sampling and simultaneously optimize two networks with identical architecture: a coarse network and a fine network. We fix the number of sample points on a ray to 64 for both coarse and fine sampling. Discussion on the number views can be seen in R#2 Q6.\n\n**Q2**: The article claims that only 2D supervision is required, but it does not explain how the SDF (signed distance function) is learned. If following the training pipeline in NeuralRecon or Mononeuralfusion, then this claim of \"only needing 2D supervision\" is incorrect.\n\n**A**: By ‘2D supervision’, we refer to 2D semantic supervision since we focus on semantic segmentation. Please be noted that we use’ 2D semantic supervision’ in the abstraction and introduction of our main paper, and only use ‘2D supervision’ in the experiments section occasionally for brevity. We will modify this to avoid confusion in the final version. For learning SDF, we can either use 3D supervision [37] or 2D supervision [52,c,d]. NeuS [c], SparseNeuS [d], and Mononeuralfusion [52] utilize Eikonal loss [e], which regularizes the gradient of SDF to be 1, and a rendering loss to learn SDF. MonoSDF [f] combines both Eikonal loss, and normal and depth predicted from pretrained models to learn SDF with only 2D supervision. Following these methods, it is therefore feasible to implement our method with solely 2D supervision.\n\n**Q3**: Can the network learn geometry information solely based on equation (6)? If so, why? Is there an intuitive way to explain this?\n\n**A**: Yes, we have done this experiment when only using semantic loss, but the network only can learn very coarse geometry information and this will cause the performance to drop. The reason that semantic loss can learn geometry information is that semantic maps can be seen as images with less texture to a certain extent, as shown in Fig. 1 in supplementary material. From the semantic maps in Fig. 1 in supplementary material we can see different objects with outlines, but no details.\n\n**Q4**: Why does Ours-1000 perform worse than Ours-100?\n\n**A**: Ours-1000 performs worse than Ours-100 (87.8 v.s. 93.3) on novel views, but much better than Ours-100 (87.8 v.s. 93.3) on novel scenes. This means that the model trained with more scenes Ours-1000 has better generalization capacity compared with ours-100. The reason is that it is easier for the model to overfit when training on 100 scenes compared to 1000 scenes. Consequently, Ours-100 performs better on novel view synthesis for training scenes, while performs worse on novel testing scenes. Please be noted that the ‘novel view’ refers to testing on the same scenes as training but different views, and ‘novel scene’ means that testing on different scenes from training. \n\n[a] Mingxing Tan, etc. MnasNet: Platform-aware neural architecture search for mobile. In CVPR, 2019.\n\n[b] Haotian Tang, etc. Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution. In ECCV, 2020.\n\n[c] Peng Wang, etc. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. NeurIPS, 2021.\n\n[d] Xiaoxiao Long, etc. SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views. ECCV, 2022.\n\n[e] Amos Gropp, etc. Implicit geometric regularization for learning shapes. ICML, 2020.\n\n[f] Zehao Yu, etc. MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction. NeurIPS, 2022.\n"}, {"review_id": "BHD903uBlz", "reviewer": "Reviewer_jaaU", "summary": "The paper combines \"generalisable\" neural radiance fields and semantic radiance fields. It does so in a relatively straightforward manner, borrowing and combining ideas in these two areas, and contributing a few technical advances on top. It demonstrates good segmentation accuracy both for producing the 2D segmentation of new views and to extract 3D segmentations of the underlying scene geometry.", "questions": "* Is the \"generalisable\" radiance field actually faster or less data-hungry than test-time optimisation when applied to novel scenes at test time?\n\n* The \"generalisable\" radiance field seems to take a significant hit in accuracy when applied to new scenes at test time. How does the accuracy compared to performing test-time optimisation (i.e., semantic NerF) on those?\n\n", "limitations": "* There is no section dedicated to discuss limitations (even though the latter is recommended in the guidelines). More importantly, I could not find any substantive discussion of the limitations at all.\n\n* There is no discussion of societal impact, but I don't believe this is required for this paper.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper combines the benefits of \"generalisable neural fields\" and semantic neural fields, essentially inheriting the advantages of both. Compared to prior generalisable neural fields, it adds a semantic dimension, which allows to fuse 2D segmentations into the 3D reconstruction. Compared to semantic neural fields that use test-time optimisation, it can in principle be faster and use a smaller number of images.\n", "weaknesses": "*   Some key aspects of the empirical validation seem to be left untested. The main point of a \"generalisable\" radiance field compared to a radiance field optimised at test time is to be either (1) faster and/or (2) less data hungry. However, I do not believe the experiments test any of this. Table 1 shows that the proposed \"generalisable\" field  is comparable (but worse) than semantic NeRF (optimisation based) on the *training scenes*. In a training phase, the \"generalisable\" field cannot should not provide any advantage in terms of speed or data economy compared to just fitting individual scenes.\n\n    The same Table 1 shows generalisation to new scenes, which would be the main point of the \"generalisable\" field, but this experiment:\n       1. does not compare the accuracy obtained on the new scenes to performing test-time optimisation on the same scenes;\n       2. It does not show whether there is a speed advantage compared to test-time optimisation;\n       3. It does not show if there is a dat economy advantage compared to test-time optimisation.\n    In short, we only know that the generalisable field works to some extend on new scenes, but we have no idea how that compare to test-time optimisation on those scenes, which would be the obvious alternative.\n\n* In Table 1, Ours-100 performs *better* than Ours-1000 (using 10x more data) on the training scenes. This is not entirely unexpected, but it strongly suggests that the \"generalisable\" neural fields take a significant hit compared to performing per-scene optimisation. This makes it even more important to check this gap on the new scenes. At the very least, the authors should discuss this aspect of the table in some depth.\n\n* line 272: The method applied on new scenes is found to outperform Mask2Former. If I understand the method correctly, thought, it is based on fusing/warping/reweighing the Mask2Former predictions using Eq. (5), even at test time. It is not the model by itself that outperforms Mask2Former, but it is the combination of the model with Mask2Former which is better than Mask2Former alone. It may be a good idea to reinforce this point at line 272.\n\n* The comparison in Table 2 is, for the most part, indicative. It compares different methods which take different kinds of  inputs and are supervised in very different ways. For example, it sounds impressive that this method can be competitive with ScanComplete (2018) that takes as input 3D ground truth, but then again the latter performs segmentation based on an (incomplete) LiDAR point cloud, whereas this method uses Mask2Former segmentations form 2D images of the scene -- hardly comparable.\n\n* It is unclear how the model is trained. The authors do not mention any appearance / RGB loss, so it seems that all the various networks are trained purely with an objective function involving the prediction of segmentation masks. Is this the case, or do you also employ an appearance loss as in most prior \"generalisable\" radiance fields?\n\n* The goal of the paper, namely combining \"generalisable\" and semantic neural fields, is a bit incremental. There are elements of novelty in the specific technical details of the combination, but there aren't big surprises.\n\n* Minor issues:\n   * line154: the \"variance\" $\\mathbf{v}$ of the feature vectors $\\mathbf{F}_i$ should be a matrix, not a vector of the same dimension. Do you mean that this is the diagonal of the covariance matrix?\n   * line 156: why do you call $\\mathbf{B}$ a *cost volume*? These are primarily just features (the only thing related to the idea of cost is the variance term, the computation of which could be weakly interpreted as comparing features from different views).\n   * line 178: $\\Delta \\mathbf{d}_i$ is a vector, not a distance.\n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Respose to Reviewer jaaU (R#5)\n\nThanks for your reviews. We are not sure of the meaning of “test-time optimization” mentioned repeatedly in the reviews. We reckon that this refers to “per-scene optimization” and provides our clarification based on this understanding. \n\nWe would also like to clarify the difference between _per-scene optimized NeSF_ and _generalizable NeSF_. For per-scene optimized NeSF, we need to train a semantic NeRF for several hours on each new scene on training data with fully annotated semantic labels. In contrast, the aim of generalizable NeSF is to learn a semantic NeRF that can perform inference on new scenes _without_ further training. Furthermore, generalizable NeSF does not require training data with fully annotated semantic labels for each new scene. Generalizable NeSF is thus more practical than per-scene optimized NeSF because it can apply to new scenes directly without annotating and training.\n\n**Q1**: Compare the accuracy obtained on the new scenes to performing test-time (per-scene) optimisation on the same scenes.\n\n**A**: In our generalizable NeSF setting, there is no annotated training data for novel scenes.  We therefore do not perform per-scene optimization on novel scenes in Tab. 1 since it would be an unfair comparison. Nonetheless, we note that the performance of per-scene optimized NeSF is similar to Novel View in Tab. 1 (about 94%) when annotated data is available for training on novel scenes.\n\n**Q2**: Is there a speed advantage compared to test-time optimisation?\n\n**A**: Yes. Per-scene optimized NeSF needs to annotate labels and optimize first and then perform inference. The annotation and training cost a lot. For example, ignoring the time of annotation, Semantic NeRF [51] needs 10 hours to finish optimizing on a new scene, but our method can infer directly (~4 minutes). This is a 150x speedup.\n\n**Q3**: Is there a data economy advantage or less data-hungry compared to test-time optimisation?\n\n**A**: Compared with per-scene optimized NeSF that requires training on annotated novel scenes, our method performs inference directly on a new scene _without_ training and _does not_ require any semantic annotations.\n\n**Q4**: Why is Ours-100 better than Ours-1000 on the training scenes?\n\n**A**: Ours-1000 performs worse than Ours-100 (87.8 v.s. 93.3) on novel views, but much better than Ours-100 (87.8 v.s. 93.3) on novel scenes. This means that the model trained with more scenes Ours-1000 has better generalization capacity compared with ours-100. The reason is that it is easier for the model to overfit when training on 100 scenes compared to 1000 scenes. Consequently, Ours-100 performs better on novel view synthesis for training scenes, while performs worse on novel testing scenes. Please be noted that the ‘novel view’ refers to testing on the same scenes as training but different views, and ‘novel scene’ means that testing on different scenes from training.\n\n**Q5**: It is fusing the Mask2Former predictions instead of the model by itself that outperforms Mask2Former.\n\n**A**: We apologize for the confusion, we will clarify it in the final version. Indeed, in Line 272 of our paper, we refer to “our method” as the fusion of the 2D semantics predicted by the pretrained Mask2Former, which is called a soft voting mechanism in the paper. Specifically, our soft voting mechanism remedies inconsistencies across multiple views of 2D semantic segmentation via the learning of proper weights, as shown in Fig. 1 of the attached PDF file in the rebuttal. \n\n**Q6**: The comparison in Table 2 is indicative. It is unnecessary to compare with methods with different inputs and supervisions.\n\n**A**: We respectfully disagree. The comparisons in Tab. 2 are with methods trained on 3D semantic supervision while our method is at a disadvantage of being trained on only 2D semantic supervision. Nonetheless, we are still able to show comparable performance with these methods. \n\n**Q7**: Does this method need RGB loss?\n\nA: We do have the RGB loss, but omitted it due to our focus on the semantic field. We will add the RGB loss into the supplementary of our final paper. More details of the training can be found in R#4Q1.\n\n**Q8**: GNeSF is a bit incremental.\n\n**A**: We respectfully disagree. Unlike other semantic NeRF methods that simply add a semantic head, our main contributions are the novel soft voting scheme and visibility module to learn a generalizable neural semantic field. With the proposed modules, our method obtains SOTA on both semantic view synthesis and 3D semantic segmentation. On 3D semantic segmentation, our approach with only 2D annotations can even outperform existing strong supervision-based approaches.\n\n**Q9**: Line154: the \"variance\" $v$ of the feature vectors $F_i$ should be a matrix.\n\n**A**: The \"variance\" $v$ is a vector. It is per-channel variance across views following SparseNeuS [a].\n\n**Q10**: Why do you call $B$ a cost volume?\n\n**A**: We follow SparseNeuS [a] to call $B$ a cost volume. To avoid ambiguity, we will call it feature volume in the final version.\n\n**Q11**: $\\Delta d_i$ is a vector, not a distance.\n\n**A**: $\\Delta d_i$ is indeed a vector, and we bold it in the article to show that it is a vector. To avoid ambiguity, we will call it displacement in the final version.\n\n**Q12**: GNeSF takes a significant hit in accuracy when applying to new scenes. How about the accuracy compared to performing per-scene optimized NeSF?\n\n**A**: Per-scene optimized NeSF does not have generalization ability and thus cannot infer a new scene without training on annotated data. Refer to Q1 for more details on training on a new scene.\n\n**Q13**: Discuss limitations and societal impact.\n\n**A**: The answers can be found in R#1Q6.\n\n[a] Xiaoxiao Long, etc. SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views. ECCV, 2022.\n"}]

Author rebuttal:
There are figures and tables in the PDF. In Fig. 1, we show some successful and failure cases and explanations can be found in R#1Q7. Tab. 1 shows the comparison with other generalizable NeSF and explanations can be found in R#2Q2. Tab. 2 shows the effect of the number of nearby views and explanations can be found in R#2Q6. Tab. 3 exhibits the integrated ablations including soft voting and explanations can be found in R#2Q7. 

Meta-review: This paper studies generalizable neural semantic fields that allow for 3D semantic prediction on novel scenes without test-time optimization. While the initial reviews are mixed, the authors provide a strong rebuttal that convinces most of the reviewers to be positive. Agreeing with most of the reviewers, the AC believes this work makes good technical progress and is worth publishing.
Decision: Accept (poster)