NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes
Authors: Suhani Vora
Noha Radwan
Klaus Greff
Henning Meyer
Kyle Genova
Mehdi S. M. Sajjadi
Etienne Pot
Andrea Tagliasacchi
Daniel Duckworth
Venue: Accepted by TMLR ()

Abstract: We present NeSF, a method for producing 3D semantic fields from posed RGB images alone. In place of classical 3D representations, our method builds on recent work in neural fields wherein 3D structure is captured by point-wise functions. We leverage this methodology to recover 3D density fields upon which we then train a 3D semantic segmentation model supervised by posed 2D semantic maps. Despite being trained on 2D signals alone, our method is able to generate 3D-consistent semantic maps from novel camera poses and can be queried at arbitrary 3D points. Notably, NeSF is compatible with any method producing a density field. Our empirical analysis demonstrates comparable quality to competitive 2D and 3D semantic segmentation baselines on complex, realistically-rendered scenes and significantly outperforms a comparable neural radiance field-based method on a series of tasks requiring 3D reasoning. Our method is the first to learn semantics by recognizing patterns in the geometry stored within a 3D neural field representation. NeSF is trained using purely 2D signals and requires as few as one labeled image per-scene at train time. No semantic input is required for inference on novel scenes.

Published in Transactions on Machine Learning Research (July/2022)
NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes
Suhani Vora* Google Research
svora svorglggooo..com
Noha Radwan' Google Research
Klaus Greff Google Research
klousg@
Henning Meyer
lutmann@google.coo
Google Research
Kyle Genova Google Research
ggemova@google.. com
Mehdi S. M. Sajjadi Google Research
mssijodi@google.com
Etienne Pot Google Research
epot@google.comm
Andrea Tagliasacchi
atagliasacchi@ggogle.com
Google Research, Simon Fraser University
Daniel wochworth
duckuorthd@google.com
Google Research
Reviewed OpenReview: http::///gpaaaeerse..eeee//rrrrr
Abstract
We present NeSF, a method for producing 3D semantic fields from posed RGB images alone. In place of classical 3D representations, our method builds on recent work in neural fields wherein 3D structure is captured by point-wise funetions. We leverage this methodology to recover 3D density fields upon which we then train a 3D semantic segmentation model supervised by posed 2D semantic maps. Despite being trained on 2D signals alone, our method is able to generate 3D-consistent semantic maps from novel camera poses and can be qucried at arbitrary 3D points. Notably, NoSF is compatible with any method producing a density field. Our empirical analysis demonstrates comparable quality to competitive 2D and 3D semantic segmentation baselines on complex, realistically-reneer scenes and significantly outperforms a comparable neural radiance field-bar based method on a series of tasks requiring 3D reasoning. Our method is the first to learn semantics by recognizing patterns in the geomelry stored within 3D neural field represenlation. NeSF is trained using purely 2D signals and requires as few as one labeled image per-scene at train time. No semantic input is required for inference on novel scenes.
* Denotes equal contribution.

---

Published in Transactions on Machine Learning Research (222))022)
2D
2D & 3D Semantics
RGB
Posed RGB
Density Field
Semantic Field
Overview We train our method on collections of posed 2D RGB images and 2D semantic maps, each collection describing an independent scene. Given a new set of posed 2D RGB images, we extract an implicit volumetric representation of the scene's 3D geometry and infer 3D semantic field. The semantic field can then be used to render dense 2D semantic maps from novel camera poses or queried directly in 3D. Our method generalizes to novel scenes, and requires as little as one semantic map per scene at training time.
Figure 1:
1 Introduction
High-level semantic understanding of 3D scenes as captured by digital images and videos is fundamental objective of computer vision. Well-studied tasks such as scene classification (Krizhevsky et al., 2012), object detection (Liu et al., 2020a), semantic segmentation (Lateef & Ruichek, 2019), and instance segmentation (Hafiz & Bhat, 2020) infer semantic descriptions of scenes from RGB and other sensors and form the foundation for applications such as navigation (Bonin-Font et al., 2008) and robotic interaction (Billard & Kragie, 2019).
The most common approach to scene understanding is to narrow the scope to 2D (image-space) reasoning, wherein classical imagc-to-image architectures (Ronneberger et al., 2015) are trained on large collections of semantically -allly-nnoottted images (Lin et al., 2014). These methods, while straightforward to apply, only produce per-pixel annotations and largely ignore the underlying 3D structure of the seene. Instead, our goal is to use a set of RGB images with known poses to produce a 3D semantic field: function mapping 3D positions to probability distributions over semantic catogorics.
For 3D semantic segmentation, most prior work relies on 3D sensors (Tzadi al., 2011; Douillard al., 2011; Kowdle et al., 2018) or 3D semantic labels (Dai al., 2017; Caesar et al., 2020a). Convolutional network architectures have been designed for 3D point clouds (Qi et al., 2017a; Choy et al., 2019), voxel grids (Tran et al., 2015), and polygonal meshes (Hanocka et al., 2019; Masci et al., 2015). However, 3D sensors are not affordable nor as widely available as RGB cameras, and 3D) annotations are significantly more challenging to produce than their 2D counterparts and are hence generally scarce (Genova et al., 2021). Conversely, 2D sensors are ubiquitous. Therefore, the ability to reason about 3D information from 2D supervision alone is crucial to the deployment of 3D computer vision scale.
To overcome the aforementioned challenges, researchers have adopted hybrid 2D/3D reasoning, so as to propagate densely supervised semantic signals from 2D projections to underlying 3D substrate (Dai & Nießner, 2018; Genova et al., 2021; Kundu et al., 2020). At test time, these methods still require a classical 3D representation to be provided as input, hence limiting their applicability and performance. One interesting exception Atlas (Murez et. al., 2020), which only requires posed photos at test time but still requires 3D supervision to train.
In parallel to these developments, a new family of 3D representations have emerged based on coordinate-ver functions (Xie et al., 2021). In this regime, trains neural network to predict quantities such occupancy,
2

---

Published in Transactions on Machine Learning Research (July /222)
signed distance, density and radiance. However, the vast majority of methods built on this approach target computer graphics applications such as novel view synthesis (Wang et al., 2021a; Zhang et al., 2021; Liu et al., 2020b; Rebain et al., 2020; Yu et al., 2021c) without an interpretable semantic understanding of the scene. One notable exception is Semantic-NeRF (Zhi et al., 2021), which regresses a per-3D point semantic class in addition to radiance and density. Semantic-NeRP produces novel views given a set of inpul views for a scene. Generalization to novel scenes is achieved by providing pscudo-labeled 2D semantic maps as input, which are fused by NeRF--like model (Mildenhall et al., 2020) to produce labeled novel views. Generalization across scenes is thus consequence of the 2D CNN that is used for producing pseudo labels. While this scales efficiently, it biases the model to the same pitfalls as 2D CNN methods where texture and color are more prevalent than shape information for segmenting objects (Geirhos et al., 2018).
In this work, we introduce Newral Semantic Fields (NeSF), a method for semantic segmentation of 3D scenes via image-space semantic annotations; see Figure 1. In place of an explicit 3D representation, NcSF builds on the implicit representations of geometry recovered by methods such as NeRF. In particular, we apply a neural network to NeRF's densily field to predict. what we refer to as scene's semanlic field. A semantic field is thus defined as coordinate-wis function mapping 3D points to probability distributions over semantic categorics. Similar to Semantic-NeRF, the closest contemporary method, we apply the volumetric rendering equation to generate 2D semantic maps, enabling supervision from posed semantic annotations in image-space. Unlike Semantic-NeRF, NeSF's 3D segmentations are derived directly from a scene's 's inferred 3D geometry. To the best of our knowledge, NeSF is the first method capable of producing dense 2D and 3D segmentations of novel scenes from only posed RGB images.
As large scale datasets of 3D semantically annotated scenes with sufficient high quality RGB views are scarce, propose three novel datasets of increasing complexity: KLEVR, ToyBox5, and ToyBox13. While datasets for 2D and 3D semantic scene understanding already exist (Dai et al., 2017: Chang et al., 2017; Straub et al., 2019; Caesar et al., 2020b), they presently lack the scale, detail, and precision necessary to to produce high-quality NeRF reconstructions. In particular, ScanNet suffers from issues with motion blur and exposure variation, MatterPort3D panormas are loo sparsely captured, and Replica contains too few scenes. Though early attempts to resolve issues such as image blur (Ma et al., 2021) have been made, we consider the additional method development required to succossfully train NeRF representations on such scenes as beyond the scope of this work. We construct over 1,000 scenes of randomly-place render hundreds of RGB images with realistic lighting and materials. Notably, random object placement breaks relational consistencies between objects that exist in available datasets (e.g. chairs are likely to be found in the neighborhood of tables), adding an additional level of difficulty. Each RGB image is paired with corresponding ground-truth camera intrinsics and extrinsics, a semantic map and a depth map. We evaluate our method on these three datasets and compare its performance to competitive techniques in 2D and 3D scene understanding.
objects and
Contributions.
We introduce the first method for generating 3D semantic fields from the geometry of neural fields, trained solely on posed RGB images and semantic maps. Unlike prior work, our method (i) can be queried anywhere within a bounded 3D volume, (ii) is capable of rendering semantic maps from novel camera poses, and (iii) generalizes to novel scenes with as few as one semantic map per scene at training time.
We propose three new synthetic datasets for 2D and 3D semantic scene understanding. In total, these datasets are comprised of over 1,000 scenes and 3,000,000 realistically-render frames. We release these datasets along with code to reproduce them to the community upon publication.
and semantically annotated
2
Related works
We now briefly overview related work in semantic segmentation (Minaee et al., 2021; He et al., 2021) and 3D reconstruction (Jin et al., 2020).
Semantic segmentation. Semantic segmentation is a heavily rescarched area, with most methods targeting a fully supervised, single modality problem (2D: Minace et al. (2021) or 3D: He et al. (2021)). 2D approaches like DeepLab (Chen et al., 2017) train CNN to segment each pixel in an image. There are also analogous
3

---

Published in Transactions on Machine Learning Research (222)
600
six]
O(N)
rou imagss yingle scene)
Danaty Held
Ord (sampled))
33 Litt P.D. meseoning
HeatU Grd (strplldd)
MJP
Semanto Meps (3D supariitiont
(pre-trrined
(somentic
Fendeting
Figure 2: Architecture - Given a pre-trained NeRF model, we sample its volumetric density grid to obtain the 3D scene representation. This grid is converted to a semantic- grid by employing a fully convolutional volumo-to-volune network thus allowing for geometric reasoning. The cemantic-feature grid is in turn translated to semantic probability distributions using the volumetric rendering equation. Note the semantic 3D UNet is trained across all scenes in the TRAIN SCENES set, though not explicitly depicted for the sake of simplicity, Additionally, note that NeSF is trained solely using 2D supervisory signals and that no segmentation maps are provided at test time.
approaches 3D for various shape representations point clouds (Qi et al., 2017b; Hu et al., 2020), sparse or dense voxel grids (Dai & Nießner, 2018; Graham al., 2018; Han al., 2020), or meshes (Hanocka et al., 2019; Huang al., 2019). In contrast these methods, our method reconstructs and then segments dense 3D representation from 2D inputs and superviston alone, and does not require ground truth 3D annotations or input geometry.
Hybrid and multi-modal methods. Many methods use one data modality to supervise or inform another (Ephrat al., 2018; Koepke et al., 2020; Alwassel et al., 2020; Nagrani al., 2020; Gupta al., 2016; Jing et al., 2020). For 3D semantic segmentation, multiview fusion (Hermans et al., 2014; Kundu et al., 2020; Armeni et al., 2019; McCormac al., 2017; Mascaro et. al., 2021; Zhang et al., 2019) is popular family of methods that requires only image supervision. However, these methods reason exclusively in the image domain and require an input 3D substrate such as a point cloud or polygonal mesh on which to aggregate 2D information. Similarly, Genova et al. (2021) propose a method for 3D point cloud segmentation from 2D supervision, but still require input 3D geometry. In an independent line work, researchers have proposed pipelines for 3D segmentation that benefit from 2D image features (Kundu et al., 2020; Jaritz al., 2019; Dai & Nießner, 2018; Vora al., 2020). Unlike our method, these approaches also require full 3D supervision.
Implicit representations. Similar to our approach, Atlas (Murez et al., 2020) simultaneously learns a 3D implicit TSDF reconstruction and its scene segmentation from posed 2D images. However, this approach requires ground truth 3D data and supervision, while our method requires only images at both train and test time. Other methods use implicit representations to reconstruct a 3D scene (Sucar et al., 2021; Peng et al., 2020) or shape (Chen et al., 2020; Deng et al., 2020; Mescheder et al., 2019; Chen & Zhang, 2019; Niemeyer et al., 2020; Peng, 2021; Genova al., 2019; Oechsle et al., 2021; Park et al., 2019; Takikawa, 2021). A more recent work approaches the problem with image supervision alone (Sucar et al., 2021) but does not consider semantics.
Neural radiance fields. Recently, a variety of methods based on NeRF (Mildenhall et al., 2020) have become popular for novel view synthesis (Martin-rruall et al., 2021; Barron al., 2021; Jeong, 2021; Lin et al., 2021; Wang et al., 2021a), 3D reconstruction (Rematas et al., 2021; Yariv et al., 2020; Jain et al., 2021; Jang & Agapito, 2021: Henzler, 2021; Yu et al., 2021c; Yuan et al., 2021; Chen & Xu, 2021), generative modeling (Meng, 2021; Schwarz et al., 2020; Niemeyer & Geiger, 2021) and semantic segmentation (Zhi et al., 2021). The majority of these models demonstrate impressive results but are only applicable in the single-scene setting. Other methods generalize to novel scenes but focus on novel view synthesis or reconstruction. Most. relevant to our method is SemanticcNeRRF (Zhi et al., 2021). Using ground-truth or pseudo-labele semantic maps, it leverages the multi-view fusion capabilitics of NeRF to generate 2D semantic predictions from novel views. In contrast to prior work, our approach directly generalizes novel scenes using the 3D geometry extracted from the neural fields.
4

---

Published in Transactions on Machine Learning Research (July/2022)
3
Method
We train NeSF on a collection of S scenes, each described by a collection of RGB images {C E [0, 1| and paired to collection of semantic maps {Sg, E x Both images and maps indexed by camera assume that each RGB map is paired with a
index and scene index For the sake of exposition, semantic map and that cach scene contains C map pairs, but the method itself makes no such assumption. Similar to prior work, we also assume the availability of camera calibration parameters {Te,c E RT} providing an explicit connection between each pixel and the 3D ray r cast within the 3D scene. We consider the problem of jointly estimating the camera calibration and the scene representation outside the context of this work, see recent approaches (Wang et al., 2021b; Lin et al., 2021; Yen-Chen et al., 2021). Our method involves two stages, which are described in the following subsections:
Section 3.1: In the first stage, we pre-train neural radiance fields on posed RGB maps {(Cg) Ve.o)] independently for each scene s E S]. This results in set of neural radiance fields with network parameters (0,)- To focus the core task of understanding from 3D geometry,
disregard the radiance portion of these fields and employ the volumetric density fields [x|10,) (0)) E [0, oo) below.
translation network T parameterized by T = = [Tueet: } Given scene's 3D geometry represented by density field o, o- -- (0)): this network produces a 3D semantic field s(x T) assigning each point a probability distribution over semantic categories. While the translation network produces a 3D field, we apply the volumetric rendering equation obtain 2D semantic maps from reference camera poses (7x.c). Predicted semantic maps can then be compared to their ground truth counterparts {Setc } in differentiable way.
Section 3.2: In the second stage, train density-to-semanlle
3.1 NeRF pre-training
To extract an accurate, dense representation of each scene's 3D geometry, we leverage neural radiance fields as proposed in Mildenhall et al. (2020). To simplify notation, we drop scene index s for the remainder of this section as all scenes can be trained independently and in parallel. More specifically, given a collection of posod RGB images {(Cgs ())), and denoting with ~ R(77) rays corresponding to pixels from image Ca", neural radiance field model with parameters 0 is trained by minimizing the squared photometric reconstruction loss:
Ligh (0) = =E B [[[C(r (0) C (r)
(1)
where (r) is the ground truth color of ray passing through a pixel in image e, and the color C(r 0) is computed by applying the volumetric rendering equation with the ray's 's near and far bounds & E Ly]:
C(r = L/
wct c(1 (1)
(2)
Let r(t)=o+ + td represent point along ray with origin o and direction d. Weight w(t==w(r(t)) is then defined as:
w(() = exp (-f,otu) du
(1)
visibility r(t) from density r(t)
where the volumotric density (t) and radiance fields c(t) are predicted by a multi layer perceptron (i.c. MLP) with Fourier feature encoding. We refer the reader to the original work (Mildenhall et al., 2020) for further details and the discretization of these integrals (Max, 1995).
Training. While neural radiance fields are acknowledged to be slow to train, we find that we are able to fit single model to sufficient quality in =220 minutes on cight TPUv3 cores on the Google Cloud Platform. Once trained, the per scene parameters )0.) are held fixed. While we acknowledge the slow training speed of neural radiance fields, we utilize as an initial prooeetf-concept for the proposed method. Within the research community, several efforts are targeting efficiently speeding up and scaling NeRF-like models (Müller et al., 2022; Yu et al., 2021a). We view extending NcSF to utilize information generated by such methods as future work.
5

---

Published in Transactions on Machine Learning Research [July/2022]
3.2 Semantic Reasoning
We now present a method for mapping 3D density fields to 3D semantic fields. To recap, we train translation model T(o T) to produce a semantic field s(x| o,, being given access to the density field of a scene o, where s assigns a probability distribution over semantic categorics at cach 3D point in space. We optimize translation model's parameters T with 2D annotations alone. Our inspiration is drawn from methods that translate explicit representations of geometry into semantics (Qi al., 201 7a; Dai et al., 2018), and observing that density fields provide an implicit notion of geometry. For ease of exposition, the overall architecture illustrated in Figure 2, is broken down into several discrete steps: ! density grid extraction, spatial reasoning, 3 feature decoding, @ supervision, and @ data augmentation.
Density grid extraction. Our method starts by uniformly evaluating the density field on a 3D lattice with spacing 6 between samples:
E(o,) = (x s.t. x E c-1:c: +1]})
(4)
While this operation limits the spatial resolution of the original density field, presents a natural representation for further processing.
Spatial reasoning (3D). We apply a 3D UNet (Çiçek et al., 2016) to Σ, to obtain a fealure grid F, of the same spatial resolution as Σg:
F(2s Tunet) = UNet3D(L2 Tuuet)
(5)
This step is essential as a point-wisc measurement of the density field (x) does not contain sufficient information to capture 3D structure. After all, (x) only measures the volumetric density at point, while the 3D structure requires reasoning over local spatial neighborhoods. Note that we share translation network parameters Tunet across scenes, thereby enabling the network to learn features across scenes well facilitating gencralization to novel scenes without needing pre-labeled images in input.
Feature decoding. Given a query point x E R we interpolate within the feature grid F, to obtain feature vector corresponding x. We then employ a neural network decoder D generate a field of probability distributions over semantic categories:
s(x F., mmpp) = F( T
(6)
where D is multilayer perceptron with trainable parameters Tmlp and TriLerp applies trilinear interpolation similar to (Liu et al., 2020b). Like their UNet counterpart, parameters Tmlp are shared across all scenes.
Supervision. To supervise the training of parameters т, we employ volumetric rendering as NeRF (Milden- hall al., 2020) but adapt it render semantic maps as in Semantic-NeRF (Zhi et al., 2021):
I' wi(| o.) s(x|o,,, o T) dt
S(r (tllo,,)) T)
(7)
We supervise the training process for by minimizing the softmax cross-entropy between rendered semantic and ground truth semantic maps:
Lsem (T) = E, [Ceem (0., T) where:
(8)
Lyem (6,, = > En-R(re) [CE(S(r] (CE)
We include an additional smoothness regularization term to encourage similar predictions in local neighbor- hoods. We sample points x Uniform([-1, 1]1) and normally distributed noise ~ A0(0, 0.01),
Lreg( Exx, [||s(x|[, 7) s(x + e(F,
(9)
Our total loss is thus L(T)= L + >XCrog(T).

---

Published in Transactions on Machine Learning Research (July/2022)
Table 1:
Dataset statistics Each dataset consists of set of train and novel scenes, wherein each scene's cameras are split into a train and test set (denoted by * */").
KLEVR
ToyBox5
ToyBox13
# scenes
100 / 20
500 / 25
500 / 25
cameras/scene
210 90
210 90
210 / 90
# total cameras
36,000
1,575,00
1,575,00
frame resolution
256 x 256
256 x 256
256 x 256
# objects/scemt
4-12
4-12
4-12
# object instances
5
25,905
39,695
# background instances
1
383
383
Data augmentation. To increase the robustness of our method and similarly to classical methods (Qi et al., 2017a;b), we apply data augmentation in the form of random rotations around the z-axis (i.e. upwards). In particular, randomly sample an angle [0, 2x) at each step of training. Rather than extracting density grid in NeRF's original coordinate system, we construct rotation transformation Ry and query NeRF at points x' = (x), resulting in the following density grid:
Σ, (o 0.) s.t. X R, ([:
(10)
Subsequently, the spatial reasoning applied to the rotated density field to produce feature grid. To query the feature grid, the ray of interest is similarly rotated in order to sample semantic features appropriately. The remainder of the algorithm remains as before. Note that this procedure does not necessitate the retraining of NeRF models.
4 Datasets Table 1 and Figure 3
To investigate NeSF, we require datasets describing the appearance and semantics of a large number of scenes from multiple points of view. While existing datasets based on indoor and self-driving sensor captures exist (Dai et al., 2017; Chang et al., 2017; Caesar al., 2020b; Sun et al., 2020), we desire controlled setting where distractors such motion blur, camera calibration error, and object motion can be eliminated. this end, we introduce three new datasets built on Kubric (Greff et al., 2022): KLEVR, ToyBox5, and ToyBox13. Each dataset consists of hundreds of synthetic scenes, each containing randomly-placed 3D objects which are realtt-rall rendered by a path tracer (Blender, 2018) supporting soft shadows, physically based materials, and global illumination effects. Each scene described by a set of posed frames, where each frame provides an RGB image, semantic map, and a depth map rendered from a shared camera pose. We provide the Kubric script to generate such scenes, so as to enable follow-up research to build increasingly challenging datasets. Figure 3 depicts images from sampled scenes of the generated datasets.
To
KLEVR. We design the KLEVR datasct to be a simple testbed akin to MNIST in machine lcarning. Inspired by CLEVR (Johnson et al., 2017), each scene contains to 12 simple geometric objects placed randomly on a matte grey floor. Each object assigned random hue and scale, and is constrained to lie within a fixed bounding box. The semantic category of an object is set equal to its geometry class (e.g. cube, cylinder, etc). While only the shape of each object is semantically relevant, color, scale, and placement serve distractors. For each scene, we render 300 frames from randomly-sampled camera poses aimed at the scene's origin. Camera poses are constrained to lie in the upper hemisphere surrounding the scene. For each frame, we rendor an RGB image, a semantic map, and a depth map; the latter is solely used by some of the compared baselines, and for computing evaluation metrics.
ToyBox5 and ToyBox13. These datasets designed to imitate scenes of children's bedrooms and are designed to be more challenging. Scenes are constructed from a large vocabulary of ShapeNet (Chang et al., 2015) objects coupled with HDRI backdrops (floor, horizon, and environment illumination) captured in the wild (Zaal et al., 2021). . Like KLEVR, each scene consists of 4-12 randomly-placed objects, and frames are seeeenddntly--saaple
iwward-facing camera poses. Objects are sampled at random from the 5 and 13 most common object categorics, respectively for ToyBox5 and ToyBox13. Such splits have been commonly used in the 3D deep learning literature (Groueix et al., 2018; Mescheder al., 2019; Genova et al., 2019; Deng et al., 2020). Like the objects themselves, backdrops are sampled at random when constructing
rendered from 300
7

---

Published in Transactions on Machine Learning Research (July/2022)
Scene Visualization
KLEVR
ToyBox5
ToyBox13
Figure 3: Dataset examples- (Teft) The frames are rendered from independently-samplee inward-facing camera poses. (Right Each frame includes an RGB image, semantic map, and depth map (not pictured here).
scene. With thousands of objects per category to choose from, most object instances appear rarely or only once.
Train/test splits. To enable evaluation from novel views within the same scene, we randomly partition each scene's frames into TRAIN CAMERAS and TEST CAMERAS; the latter representing the set typically used to evaluate methods in novel view synthesis (Mildenhall et al., 2020). For evaluation across scenes, we further partition scenes into TRAIN SCENES and NOVEL SCENES. Table depicts the statistics for each of the proposed datasets.
5 Experiments
5.1 Setup
We evaluate NeSF on the three datasets described in Section 4. Unless otherwise stated, train NeRF models all TRAIN CAMERAS from all TRAIN SCENES. We provide NeSF with supervision from semantic maps corresponding to 9 randomly-chosen cameras per scene. These frames comprise approximately 4% of all RGB frames and strike a balance between multi-view information and a label-scarce regime. For 2D evaluation, we randomly select cameras from each NOVEL SCENES' TRAIN CAMERAS. For 3D evaluation, use camera parameters and ground truth depth maps to derive labeled 3D point cloud from the same cameras. Semantic segmentations are evaluated according 2D and 3D mean intersection-over-uiioo Further details regarding the training setup provided in the supplementary material.
Training Details. Each scene is preprocessed by training an independent NeRF for 25k stops with Adam using an initial learning rate of 1e-3 decaying to 5.4c-4 according to a cosine rule. Our NeRF architecture follows the original work. NeSF trained for 5k steps using Adam with an initial learning rate of 1e-3 decaying 4e-4. As input for NeSF, we discretize density fields by densely probing with e=1/32 resulting in 6 evenly-spaced points in [-1, +1}3. This density grid is then processed by the 3D UNet architecture of Çigek et al. (2016) with 32, 64, and 128 channels at each stage of downsampling. The semantic latent vector is processed by multilayer perceptron consisting of 2 hidden layers of 128 units. Our models are trained on 32 TPUv3 cores. We intend to release the code, datasets and pre-trained models upon publication.
Segmentation baselines (2D/3D) We compare NeSF to two popular semantic segmentation baselines, DeepLab (Chen et al., 2017) and SparseConwNet. (Graham et al., 2018). The aforementioned methods were
8

---

Published in Transactions on Machine Learning Research (July/2022)
TRAIN CAMERAS
TEST CAMERAS
2D mloU
3D mloU 97.8 0.7
2D mloU
3D mloU
NeSF ScmaniieNeeRPPDeeplabb)
92.7 0.2
92.6
97.5 0.7
93.7 0.002
90.0 1: 0.007
DeepLab
97.1
Semaatie-NRRF(Ground Truth)
95.3 + 0.002
93.2 + 0.009
SparsoConvNee(rruundd Truth Depth Maps)
99.7
99.7
Table 2: Quantitative comparison (KLEVR) NeSF is competitive with 2D and 3D basclines. Configura- tions marked as =_" denotes setting where methods are not applicable. As DeepLab can only be queried given an RGB input, it is not evaluated on the TEST CAMERAS. Semantic-NeRF, unlike NcSF, is trained per-scene basis, thus report the numbers on the TEST CAMERAS only. The double line separating methods is to highlight approaches that utilize different data modality compared to NeSF. trained on ground-truth 2D semantics has access to ground-truth semantic labels for all TRAIN CAMERAS. SparseConvNet utilizes 3D supervision in the form of a 3D labeled point cloud. Note that all methods here, with the exception of Semantic-NeRF, are trained using semantic supervision generated from only images per-scene.
Comantic-NeRLe
selected due to their competitive performance, and their case of application. DeepLab follows a traditional 2D semantic segmentation pipeline, producing segmentation maps from RGB images. We train DeepLab v3 with Wide ResNet (Wu et al., 2019) for 55k steps 16 TPUv3 chips. SparscConvNet is point cloud segmentation method and, unlike NeSF and DeepLab, requires explicit 3D supervision. We train SparseConvNet asynchronously on 20 NVIDIA V100 GPUs with momentum using a base learning rate of 1.5e-2 and decaying to O over the final 250k steps of training. We refer the reader to Section 5.2 and the supplementary material for further details.
Semantic-NeRi We compare NcSF to Semantic-NeRF (Zhi et al., 2021), which simultancously optimizes NeRF (Mildenhall et al., 2020) reconstruction while also fusing 2D semantic maps from the same views. While Semantic-NeRF is evaluated on 2D mIoU of unseen views, we compare with NeSF by evaluating both 2D mloU as well as 3D mloU of novel scenes. We compare Semantic-NeRF's performance with NeSF in two distinct regimes: 1) using pseudo-labeled semantic maps generated by a 2D segmentation method, and 2) using ground-truth semantic maps. While the former are easy to acquire, pseudo-labeled semantic maps may contain multi-view inconsistencies and other errors. On the other hand, is infeasible to obtain ground-truth labels for all lest scene images in practical application, and this regime serves as an upper bound Semantic-NeRF'ss performance.
Note that in each of these scenarios, one method is at disadvantage compared to the other. Training Semantic-NeRP using ground-truth semantic labels puls NeSF at a disadvantage in terms of label scarcity, as NeSF does not use ground-truth labels for NOVEL SCENES and only uses a few labeled semantic maps for TRAIN SCENES. At the same time, training Semantic-NeRF using pscudo labeled semantic maps puts NeSF at an advantage as pseudo labeled semantic maps can contain mislabeled regions which in turn affect the final method's performance. Nonetheless, Semantic-NeRF shares a number of commonalities with NeSF, report the mumbers in both regimes with the goal of presenting and understanding the limitations of each method.
5.2 Comparisons to baselines
Our first set of experiments evaluates the performance of our proposed method in comparison to baselines on the KLEVR, ToyBox-5, and ToyBox-13 13 datasets.
Lee (Chen et al., 2017) (2D). To maintain a fair comparison in 2D, we train both the semantic phase of NeSF and DeepLab on an identical set of paired RGB images and semantic maps for a fixed set of scenes (i.e. 9 per scene in TRAIN SCENES). NcSF has further access to all 210 RGB maps associated with each scene's TRAIN CAMERAS, which are used to fit per-scene NeRF models. Both methods are evaluated random sampling of frames from NOVEL SCENES (i.e. 4 per scene). To emphasize the 3D nature of
9

---

Published in Transactions on Machine Learning Research (July/2022)
ToyBox5
ToyBox13
TRAIN CAMERAS
TEST CAMERAS
TRAIN CAMERAS
TRST CAMRRAS
2D mloU
3D mloU
2D mloU
3D mloU
2D mloU
3D mlou
2D mloU
3D mloU
NeSF'
8.9
0.9
81.7 = 0.6
89.6 80.8 0.1
0.5
60.1 0.6
56.6 2 1.0
6.9
Bemantic-NeRF(Deee (DoopDa)))
72.3 0.1
56.1 +
68.2
DeepLab
81.6
63.1
Semantic-Neli
(Ground Truth)
812 0.1
93.4 0.1
81.1 0.1
91.6 = 0.1
Coneeeoonee Troth Depth Maps)
93.4
94.8
83.2
81.7
Table 3:
Quantitative comparison (ToyBox5, ToyBox13) NeSF is competitive with 2D and 3D baselines. At train time, NeSF, Semantic-NeRF and DeepLab only utilize 2D supervision. Conversely, SparseConvNet requires full 3D supervision in the form of labeled 3D point clouds. We construct oracle point clouds via back-projected depth maps, resulting in an upper bound to our method. Models are evaluated on train and test camera poses from test scenes. Configurations marked as "_" denotes a setting where methods are not applicable. Statistics for NcSF and Scmantic-NeRF are aggregated across five random initializations. Note that all methods here, with the exception of Semantic-NeRPP are trained using semantic supervision generated from only 9 images per-scene.
Fixed-Color
Mono-Color
Color-Color
Texture
2D
93.7 2.0
92.3 + 6.0 85.6 3.07
93.4 2.0
92.5 + 3.0
NeSF
3D
86.0 3.0
88.3 + 3.0
90.5 + 5.4
2D
22.7 0.0
8.0
4..0
Semantic-NeRF (DeepLab)
3D
22.0 0.0
35.2 0.0
8.1
4.07
2D
95.0 0.0
95.0 0.0
94.6 4466000 0.0
95.1 0.0
Semantic-NeRF (Ground Truth)
3D
93.7 = 0.0
92.5 0.0
94.8 + 0.0
94.4 0.0
Table 4:
Quantitative comparison on KLEVR dataset perturbations - Unlike Semantic-NeRF, NeSF
is robust to appearance and color variations in the data as it relics on geometry for reasoning about semantics.
NeSF, we evaluate additional camera poses from NOVEL SCENES where RGB information is not available. More preciscly, for every scene, we evaluate on an additional 4 TEST CAMERAS. Note that this setting is not directly comparable to DeepLab as it requires an RGB input to produce the semantic predictions.
SparseConvNett (Graham al., 2018) (3D). As SparseConvNet requires 3D input, we derive an oracle point cloud for each scene from camera poses and ground truth depth maps hence giving an unfair advantage to this method, establishing same 210 train frames used to fit NeRF models in the 2D comparison. We further select a subset of each point cloud for 3D semantic supervision; namely, the points corresponding to the 9 semantic maps supervising NeSF and DccpLab. We evaluate NcSF and SparseConvNet on two sets of 3D points on NOVEL SCENES. The first set is a subset of each point cloud correspouding to 4 randomly chosen frames from each scene's TRAIN CAMERAS. These points are available to SparseConvNett as part of each scene's 3D representation. The second set is set of additional query points derived from 4 additional frames from each scene's TEST CAMERAS. As SparseConyNet is not designed to classify points beyond its input point cloud, apply a nearest neighbor label propagation procedure to assign labels to the latter.
upper bound on performance given full 3D supervision. For this, we use the
Semantic-NeRF (Zhi et al., 2021) (2D/3D). In order to present a complete picture of the performance of NcSF comparison to Semantic-NeRF, we report numbers on two variations of Semantic-NeRF: 1) DeepLab: we use supervise Semantic-NeRF with semantic maps generated by DeepLab (Chen et al., 2017), and 2) Ground-truth: we use ground-truth 2D semantic maps. The DeepLab model used for pseudo-labeling is trained on the same set of semantic maps provided for training the semantic phase of NeSF. Specifically, the train dataset consists of 9 semantic maps per scene from cach scene in TRAIN SCENES. For Semantic-NcRF fitting on NOVEL SCENES, either the DeepLab model is used to pseudolabel all 210 RGB maps associated with each scene's TRAIN CAMERAS, or ground-trutà truth labels are supplied. Note that Semantic-NeRE is the only mothod that is trained using semantic supervision on all 210 TRAIN CAMERAS per-scene. Evaluation carriod out by evaluating on views for 4 unseen TEST CAMERAS for each of the NOVEL SCENES. Note that we do not report results for Semantic-NeRF on the TRAIN CAMERAS, as these views used to train Nemantic-NeRF
10

---

Published in Transactions on Machine Learning Research (July/2022)
for each of NOVEL SCENES. In comparison, NeSF's semantic module does not require semantic supervision at test time.
Quantitative comparisons - Table 2 and Table 3. While all methods perform comparably on the KLEVR dataset, model quality varies drastically on the more challenging datasets. On ToyBox5, our method performs comparably to DeepLab, but on ToyBox13, it underperforms by 6.6% in 2D mloU. While our method does not achieve the same level of accuracy as DeepLab on frames where RGB images are available, it is able to achieve near identical accuracy on novel camera poses, task to which DeepLab cannot be applied. Compared to Semantic-NeRF trained with pscudo-labels on ToyBox5, our method outperforms by 9.4% and 8.8% in 2D and 3D mIoU respectively. We perform comparably on ToyBox13, with a slight. improvement of 1.5% and 3.7% in 2D and 3D mloU respectively, As expected, when comparing to FeeaF trained on ground-truth labels, we notice a larger performance gap, whereby NcSF underperforms by 2.5-3.8% 8.% on ToyBox5 and 24.5-29.7% on ToyBox13. As expected, our method also underperforms SparseConvNet by 4.7-5.2% on ToyBox5 and 19.8-23.1% on ToyBox13. Unlike SparseConvNet, our method lacks access to dense, ground truth depth maps and full 3D supervision. Further, the 3D UNet architecture employed by NeSF is based on (Çigek et al., 2016), predecessor to the SparseConvNet architecture. As NeSF does not take advantage of sparsity, it must operate at a lower spatial resolution than the baseline and tends to mislabel small objects and thin structures. Though NeSF underperforms SparseConvNet today, we anticipate methodological improvements in model architecture to rapidly improve performance. Additional in-depth analysis is included in the supplementary material.
Qualitative comparisons - Figure 5 and Figure 6. Qualitatively, our method exhibits strong perfor- mance in identifying the 13 canonical categories in ToyBox13. Because our method operates directly on 3D geometry, it is not casily confused by objects of similar appearance but dissimilar geometry as demonstrated by the thin standing rifle in the top row of Figure 5. While NeSF and SparseConvNel correctly recognize the rifle's geometry, DeepLab labels it identically faces challenges with thin structures such as the tube of the standing lamp pictured in the middle row. Such structures are not well captured by the regular grids employed by DeepLab and NeSF in 2D and 3D, respectively. With access to dense, accurate point clouds, SparseConvNet correctly identifies the lamp in its entirety. One limitation particularly evident in NeSF is a tendency to smear labels across nearby objects, as demonstrated by the chair partially labeled as a display in the bottom row. Without access to appearance or fine- areiggraied geometry, NeSF is unable to identify when one object ends and another begins. Integrating appearance information and access to higher spatial resolution are potential ways to improve NcSF's accuracy.
the chair behind it. Similar to DeepLab, our method
In Figure 6, we compare the performance of NeSF and Semantic-NeRP (supervised by DeepLab pseudo-labels) all datasets. Both NeSF and Semantic-NeRF can be seen to perform well on the least challenging KLEVR dataset, as the objects can casily be segmented in either 2D or 3D. However, on the more challenging ToyBox5 and ToyBox13 datasets, we find that while Semantic-NeRR captures some fine structures, the density fields generated by SemanticNeRF contains a large amount of floaters which reduce the semantic accuracy of the model's predictions. Recently, methods such as Mip-NeREF360 (Barron et al., 2022) propose regularization terms to reduce the presence of such artifacts. Floater artifacts are best visualized in rendered videos. To view side by side comparisons of such renders, we encourage the reader to view additional results in the accompanying video from timestamp 8:07. Furthermore, on both ToyBox5 and ToyBox13, find that NeSF is more capable of capturing the fine details of the objects in the scene; like the structure of the lamp in ToyBox13 and the chair in ToyBox5. Similar to NeSF, Semantic-NeRF struggles to separate objects leading to some smearing artifacts in the semantic maps.
5.3 Ablation Studies
Our second set of experiments investigates how each component of our method affects system performance on the KLEVR dataset.
Sensitivity to features. Table 5 shows results of an ablation study where one feature of our method is varied while holding all others to their reference values. We find that each component provides a measurable improvement in 2D and 3D segmentation quality. Data augmentation in the form of random scene rotations
11

---

Published in Transactions on Machine Learning Research (July (02))
Table Ablation: NeSF hyperparameters Data augmentation, in the form of random scene rotations, increased spatial resolution of the density grid, and in- creased UNet model capacity improve 2D and 3D mloU. Experiments 25 scenes from the KLEVR dataset. Hy- perparameters correspond to the following. Random Ro- tations: whether random rotations data augmention is applicd to density grid during training, Density Grid: res- olution of the density grid along the 3 spatial dimensions as referred to by epsilon in Equation 4, UNet: number of output features for each of 3 respective CNN layers of the UNet, and MLP: respective number of (hidden layers, hidden units) per layer of the ML..
Hyperparameter
2D
3D
Random Rotations
No
81.1
75.5
Yes
92.0
97.1
(32, 32, 32)
87.5
92.1
Density Grid
(48, 48, 48)
91.2
96.0
(64, 64, 64)
91.7
89.6
(80, 80, 80)
92.0
97.1
(16, 32, 64)
89.9
94.4
UNet
(24, 48, 96)
91.5
96.4
(32, 61, 128)
92.0
97.1
(0, 32)
91.3
96.4
(1, 32)
91.8
96.9
MLP
(1, 64)
91.2
96.2
(2, 128)
92.0
97.1
Table 6: Ablation: sensitivity to recon- struction quality The accuracy our method improves with NeRF's reconstruction quality. PSNR and SSIM are are averaged across all scenes and metrics aggregated. Experiments on all scenes from KLEVR.
# RGB Images
NeRF
NeSF
PSNR
SSIM
2D
3D
5
21.2 + 1.4
0.89 + 0.02
52.2
72.4
10
24.2 1 1.2
0.92
79.6
25
00.3112
0.001
87.3
97.0
50
0.9
0.90
90.8
97.3
75
37.5
0.98
91.4
97.1
100
38.4
0.98 0.00
92.0
97.4
improves quality the most, adding 10.3% and 11.8% 2D and 3D mloU respectively. The spatial resolution of the probed NeRF density grids the second most important insufficient resolution makes smaller objects indistinguishable. We refer the reader to Figure 10 in the supplementary material for visualizations of the density grid.
Sensitivity to reconstruction quality Table 6. We investigate the robustness NeSF to NeRF reconstruction quality. To modulate reconstruction quality, we vary the number of RGB images used when fitting NeRF models from to 100. As expected, NeRF reconstruction quality measured novel views increases more RGB images provided. At the same time, find that NeSF 2D segmentation accuracy improves monolonically with NeRF reconstruction quality. Improvement of NeRF reconstruction quality is a highly active area of research, as such we anticipate methodological improvements may be directly applied to improve NcSF's performance. Surprisingly, 3D segmentation accuracy levels off near 97% when NcRF models are optimized with as little as RGB images.
Sensitivity to data scarcity Figure To investigate NeSF's applicability scenarios where labeled semantic maps scarce, investigate robustness to the number of semantic maps per scene. We find NeSF easily generalizes to novel scenes with as few maps per scene improve performance given small number of scenes, with no noticeable effect after 25 scenes. This suggests datasets consisting of videos, each with a single labeled frame, are ideal for NeSF.
semantic map per train scene. Additional semantic
We note, however, that performance does not monotonically increase with the number of semantic maps per scene, particularly in the regime of limited training scenes. While additional maps per scene will generally increase accuracy on held-out views of the training scenes, it is not guaranteed to improve performance on novel scenes.
Robustness to dataset shift - Figure 8 and Table 4
In the following, we design a set of experiments highlighting the differences between NeSF and Sernaniic-NeRP in semantic prediction novel scenes. We hypothesize that NeSF uses orthogonal information to that used by Semantio-NeRF for learning the semantics of scene. Specifically, NcSF learns semantics from 3D goometry allowing the method be robust to visual, texture based changes commonly observed amongst every day objects (Geirhos et al., 2018). A model that possesses such property is expected to perform more robustly
12

---

Published in Transactions on Machine Learning Research (July/2022
2D
3D
100
90
80
70
10
25
50
75 100
10
25
50
75
100
Figure 4: Ablation: data efficiency 2D and 3D mloU function of the number of train scenes for scenes with supervision from 1, 2, 5, 10, or 25 semantic maps per scene. NeSF generalizes to new scenes with as few as a one semantic map per scene. Additional semantic maps per scene marginally improve the accuracy. Experiments on KLEVR dataset. While performance on novel scenes is not, monotonic with respect to the number of semantic maps per scene, it is nearly monotonic with respect to number of training scenes, as expectod.
RGB
Ground Truth
NcSF (Ours)
DeepLab
SparscConvNett
Figure 5: Qualitative comparison (ToyBox13) Unlike DeepLab, NeSF is able to clearly separate objects with similar appearance but different geometry (Top). However, NeSF struggles with thin structures like lamp posts (Middle) and tends to smear labels from nearby objects (Bottom). SparseConyNet suffers from neither limitation but has access to oracle 3D geometry and full 3D supervision.
in the presence of certain dataset shifts. Accordingly, we compare NeSF to Semantic-NeRF on the following perturbations of the KLEVR dataset that contain significant shift in color or texture properties between train and test objects:
Mono-Color: TRAIN SCENES contain monochrome-colored objects, whereas NOVEL SCENES contain randomly sampled colors for each object.
Fixed-Color: Each object with the same semantic category has the same RGB color assigned to it within TRAIN SCENES. Within NOVEL SCENES, each object assigned a random color.
Color-Color: Each object within the same semantic category is randomly assigned color from a fixed set of colors at TRAIN SCENES. At NOVEL SCENES, different set of colors is used for each semantic category.
13

---

Published in Transactions on Machine Learning Research (July (
KLEVE.
ToyBox5
ToyBox13
RGB
Ground Truth
NeSF (Ours)
Semantic-NeRP
Figure Qualitative comparison on KLEVR, ToyBox5 and ToyBox13 On the KLEVR dataset, NeSF performs similar to Semantic-Ner NeRF. On both versions of the ToyBox dataset however, can that NeSF is better able to capture fine structures compared to Semantic-NeRF.
Texture-Texture: Each object within the same category randomly assigned a texture from a fixed set of textures TRAIN SCENES. Within NOVEL SCENES, a new texture used for each semantic calegory.
Figure 7 shows example images from each of the above perturbation sets.
We believe that NeSF's generalization capabilities are direct consequence of utilizing an architecture that relies on 3D geometry for learning semantics. Meanwhile, Semantic-NoRF generalizes primarily across 2D image space. Thus, expect the latter under-perform given changes in the dataset domain caused by varying color and texture.
We follow the same procedure for training our NeSF model as mentioned in 5.1. Similar to previous experiments, we train two variants of the Semantic-NeRF model; using ground-truth labels and pscudo labels from a pre-trained DeepLab model that is fine-tuned on the TRAIN SCENES.
On each of the four datasets, observe NeSF outperforms Semantic-Nelke as seen in Table On the Mono-Color dataset shift, method outperforms by 71% and 64% in 2D and 3D mloU respectively. On the Fixed-Color dataset shift, we observe difference of 54.5% and 50.4%. Semantic-NeRR's best performance is achieved on the Color-Color dataset shift, where compared to NeSF, we observe a small difference of 3.8% and 2.7% in 2D and 3D mloU respectively. In the Texture-Texte setting, NeSF outperforms Semantic-NeRP by 52% and 48.8% in 2D and 3D mloU respectively.
These experiments demonstrate the potential drawbacks of relying primarily on fusing fcatures from 2D for achieving generalization. On the other hand, leveraging the scene's is 3D geometry to learn seranttis proposed in the current model, renders NeSF robust to several types of visual dataset shift.
As the proposed dataset perturbations are challenging for 2D semantic methods, the crrors made by DeepLab propagale to Semantic-NeRF'ss predictions. This can be seen in the Fixed-Color and Mono-Color results in Figure 8, where Semantic-Neth is unable to identify simple objects effectively. This failure mode tracks on even more subtle dataset shifts such as the Color-Color dataset, where we observe an entire object incorrectly segmented by Semantic-NeRF, Conversely, in all cases, NcSF able to handle the dataset shift seamlessly demonstrated by both quantitative and qualitative results. In this work we have explored dataset shift in the
14

---

Published in Transactions on Machine Learning Research (22y)2022)
TRAIN
TEST
Mono-Color
Fixed Fixed-Coln Color
Color-Color
Texture- Texture
Figure 7: KLEVR Color Perturbations Various perturbations applied to generate color based shifts between train and test scenes.
15

---

Published in Transactions on Machine Learning Research (222)
Fixed Color
Mono- Color
Color- Color
Texture- Texture
RGB
Ground Truth
NeSF (Ours)
Semantic-NeRF
Figure 8: Qualitative comparison on different dataset pertubations By utilizing the shape information to predict the semantic category, NeSF more robust to shifts the data domain.
16

---

Published in Transactions on Machine Learning Research (July (22y)
context of visual texture. However, another common form of dataset shift includes inference data with object categories unseen during training; namely, zero-shot segmentation. Though not the focus of our work, this task is a broad and important area of study. We anticipate the same previous challenges observed and solutions developed for 2D segmentation tasks (Bucher et al., 2019) would extend to Cemantic-NRRF and NeSF as well.
6 Considerations for Societal Impact
Compute. A crucial component of the proposed method is training of a NeRF per scene in order to produce a scene representation. As our training scheme utilizes 32 cores per NeRF training run, the parallelization of training for hundreds of scenes can require thousands of cores to achieve the efficiency we have reported. Though the total number of train cycles is relatively short for a single NeRF (25,000 steps in 23 minutes), there is still the negative impact the enviromment to consider in the face of iterative experiments. Additionally, the amount of compute required to efficiently train the representations needed, may limit the scope of researchers that can readily build on our method. In order to tackle the latter aspect, we will release pre-trained NeRF models for all of the scenes we provide within this work in an attempt to make the work more accessible to the broader research community.
Privacy. We enable 3D models to be trained without the need for 3D supervision; opening up the possibility of rumning 3D inference on entirely new regime of datasets. 3D models of an environment can be obtained without the need of LIDAR sensors or 3D labelling of the data. While possibility for unforeseen use cases for example tied to surveillance data which can be easily collected in 2D, resulting in unintended negative consequences to the privacy of the general public.
it opens up the
7 Conclusions and Limitations
In this work, we present NeSF, a novel method for simultaneous 3D scene reconstruction and semantic segmentation from posed 2D images. Building on NeRF, our method is trained solely on posed 2D RGB images and semantic maps. At inference time, our method constructs a dense semantic segmentation field that can be queried directly in 3D or used to render 2D semantic maps from novel camera poses. We compare NeSF to competitive baselines in 2D and 3D semantic segmentation on three novel datasets.
In more challenging scttings, we find that NeSF underperforms competitive 2D and 3D methods but oulperforms the comparable Semantic-Nell NeRR baseline. Relative to all alternative methods, NeSF offers novel capabilities. Unlike traditional 2D segmentation methods, NeSF fuses information across multiple independent views and renders scmantic maps from novel poses. Unlike 3D point cloud mothods, NcSF operates on posod 2D information alone at both train and test time. Unlike Semantic-NeRF, NeSF leverages the 3D geometry of neural fields to learn semantics, enabling robustness visual dataset shifts in color or texture. One limitation of the current architecture is the discritization of the density field into grid. On the current set of datasets, NeSF struggles to capture small-scaled objects. We anticipate that applying NeSF in the current format to large scale scenes would be computationally inefficient due to the large emply spaces commonly present in such areas. We chose to limit NeSF to a core set of features to better explore the fundamental trade-offi and capabilities of such an approach. We further explore trade offs with respect to model choice in the supplementary material.
In addition to NoSF, we propose three new datasets for multiview 3D reconstruction and semantic sogmentation totalling over 3,000,000 frames and 1,000 scenes. Each dataset contains hundreds of scenes, each consisting of set of randomly placed objects. The more challenging of these datasets are rendered with realistic illumination and large catalogue of objects and backgrounds. These datascts along with accompanying code and pretrained NeRF models, are publicly available on our project website (to be linked upon publication).
In future work, inspired by competitive 2D and 3D baselines, we anticipate extending NeSF to incorporate 2D semantics; alike Semantic-NoRF and 3D sparsity; alike SparscConvNet will further improve accuracy. Morcover, we intend to investigate the effects of utilizing a different backbone to NeSF, with the goal of building off the improvements in the literature to tackle some of the limitations encountered by our model such
17

---

Published in Transactions on Machine Learning Research (July/2022]
as lack of ability to handle motion blur (Ma al., 2021), reducing floater artifacts and modeling unbounded scenes (Barron et al., 2022), and the high computational cost, of training NeRF models (Müller et al., 2022). While we evaluate NeSF on synthetic data due to the lack of well-suited large multi-scene real-world NeRF datasets, recent research has started tackling these problems (Zhang et al., 2022; Tancik et al., 2022), further motivating the development. of techniques for semantic segmentation of neural fields alike NeSF.
Acknowledgements
We would like to express our gratitude to Konstantinos Rematas, D. Sculley, and especially Thomas Funkhouser for their idcas and suggestions. We would like to note our doep appreciation for the project support and leadership given by Jakob Uszkoreit, without which the work would not have been possible. The authors would also like to thank Rocky Cai in his assistance in assembling the DeepLab baseline experiments.
References
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems, 33, 2020. 4
Iro Armeni, Yang He, Jun Younn Gwak, Amir R. Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE International Conference on Computer Vision, 2019. 4
Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul Srinivasan. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. In ICCV, October 2021. URL attps://arrn httpaa///aaavv..oaaaaa////// org/abs/2103. 13415. 4
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hodman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022. 11, 18, 30
Aude Billard and Danica Kragic. Trends and challenges in robot manipulation. Science, 364(6446), 2019. 2
Blender. Blender- 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http://www blender. org. 7
Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver. Visual navigation for mobile robots: A survey. Journal of intelligent and robotic systems, 53(3):263 296, 2008. 2
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. arXiv, 2019. URL http://arxiv. wwwpoogg//bbbbbbgeeeee http://a.///99/0//// 00817. 17
Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11631,
2020a. 2
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal datasct for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 11621-11631)
2020b. 3,
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint (Xtt::700.0011 2017. 3,
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-ride 3d model repository. Xix preprint ar Xiv: 1011 03012, 2015.
18

---

Published in Transactions on Machine Learning Research [July/2022]
Anpei Chen and Zexiang Xu. MVSNeRF: Fast Generalizable Radiance Field Reconstruction From Multi- View Stereo. In TCCV, October 2021. URL http:///aaxxi..rrr//////11000 15595. 4
Liang-Chiel Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions on pattern analysis and machine intelligence, 40(4))834-848, 2017. 3, 8, 9, 10
Zhiqin Chen and Hao Zhang. Learning Implicit Fields for Generative Shape Modeling. In CVPR, 2019. URL https://arxiv. http:///aaaiivvorraa//////////// / 4
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. BSP-Net: Generating Compact Meshes via Binary Space Partitioning. In CVPR, 2020. URL /ttp:p////arra.....///////11111. 16971191106691..
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolu- tional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3075 3075-3084, 2019.
Ozgün Çięek, Ahmed Abdulkadir, Soeren Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention, pp. 424 432. Springer, 2016. 6, 8, 11, 25
Angela Dai and Matthias NieBner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 452 468, 2018. 4
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annn annotte-d 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5828 5838, 2017. 2, 3, 7
Angela Dai, Daniel Ritchic, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias NicBnor. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In Proceedinggs of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 4578 4587, 2018. 6
Boyang Deng, Kyle Genova, Sofien Bouaziz, Geoffrey Hinton, Andrea Taglinsacchi, and Soroosh Yazdani. CvxNet: Tearnable Convex Decomposition. In CVPR, 2020. URL https://arxiy:: 7,7
05736.
Bertrand Douillard, James Underwood, Noah Kuntz, Vsevolod Vlaskine, Alastair Quadros, Peter Morton, and Alon Frenkel. On the segmentation of lidar point clouds. In 2011 IEEE International Conference on Robotics and Automation, pp. 2208. IEEE, 2011. 2
Ariel Ephrat, Inbar Mosscri, Oran Lang, Tali Dekel, Kevin Wilson Avinatan Hassidim, William T. Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independen. audio-visual model for speech separation. ACM Trans. Graph., 37(4), July 2018 ISSN 0730-0301. doi: 00.114/3319757..22 URL /tttps://do..rrgge
3201357. 1145/3197517 1221177777.21137.... 4
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-traince trained cmns are biased towards texture; increasing shape bias improves accuracy and robustness. Xiviv preprinl or Xio:::181 1811 12231, 2018. 3, 12
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William Freeman, and Thomas Funkhouser. Learning Shape Templates with Structured Implicit Functions, In ICCV, 2019. URL https://arxiv..rg abs/19000 06447. 4,
Kyle Genova, Xiaoqi Yin, Abhijit Kundu, Caroline Pantofaru, Forrester Cole, Avneesh Sud, Brian Brewington, Brian Shucker, and Thomas Funkhouser, Learning 3d semantic segmentation with only 2d image supervision. 3DV, 2021. 2, 4, 26
Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9224 024222222 2018. 4, 8, 10, 26
19

---

Published in Transactions on Machine Learning Research (July/2022]
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Hemming Meyer, Yishu Miso, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mchdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: scalable dataset generator. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 7
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. In CVPR, 2018. URL https://arxiv.org org/ abs/-7 7
Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2827 2836, 2016. 4
Abdul Mueed Hafiz and Ghulam Mohiuddin Bhat. A survey on instance segmentation: state of the art. International journal of multimedia information retrieval, pp. 1-19, 2020. 2
Lei Han, Tian Zheng, Lan Xu, and Lu Fang. Occuseg: Occupancy-aware 3d instance segmentation. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 2940-2949, 2020.
Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshenm: a network with an edge. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019. 2, 4
Yong He, Hongshan Yu, Xiaoyan Liu, Zhengeng Yang, Wei Sun, Yaonan Wang, Qiang Fu, Yanmei Zou, and Ajmal Mian. Deep learning based 3d segmentation: A survey. arXiv preprint arXin:2009. Xiv:2103305423, 2021. 3
Philipp Henzler. Unsupervised Learning of 3D Object Categories from Videos in the Wild. In CVPR, 2021.
URL /ttp:::///xxxvv.rr/e//
4
Alexander Hermans, Georgios Floros, and Bastian Leibe. Dense 3d semantic mapping of indoor scenes from rgb-d images. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 2631-2638.
IEEE, 2014.
Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of largo-scale point clouds. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognilion, pp. 11108 11117, 2020. 4
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, Matthias Nießner, and Leonidas Guibas. Texturenet: Consistent local parametrizations for learning from high-resolution signals on meshes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognátion, pp. 4440-4449, 2019. 4
Rui Huang, Wanyue Zhang, Abhijit Kundu, Caroline Pantofaru, David A Ross, Thomas Funkhouser, and Alireza Fathi. An Istm approach to temporal 3d object detection in lidar point clouds. In Computer ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pp. 266-282. Springer, 2020. 26
Shahram Izadi, David Kim, Otmar Hilliges, David Molyncaux, Richard Newcombe, Pushmeet Kohli, Jamic Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, al. Kinectfusion: real-time 3d reconstruction and interaction using moving depth camera. Tn Proceedings of the 24th annoall ACM symposium on User interface software and technology, pp. 559 568, 2011. 2
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis. In ICCV, October 2021. URL https://arxiv.. wttp/o/////////.o./////// 00677. 4
Wongbong Jang and Lourdes Agapito. CodeNeRF: Disentangled Neural Radiance Fields for Object Categories. In ICCV, October 2021. URL. http:///aaavvvvoo///////////////
4
20

---

Published in Transactions on Machine Learning Research (July/ (222)
Maximilian Jaritz, Jiayuan Gu,, and Hao Su. Multi-view pointnet for 3d scene understanding. In Proceedings of the IEFE International Conference on Compuler Vision Workshops, pp. 0-0, 2019. 4
Yoonwoo Jeong. Self- Salf-Caiibraaing Neural Radiance Fields. In ICCV, October 2021. URL https://arxiv. org/abs/2108 13826. 4
Yiwei Jin, Diqiong Jiang, and Ming Cai. 3d reconstruction using deep learning: survey. Communications in Information and Systems, 20(4))389-413 2020. 3
Longlong Jing, Yucheng Chen, Ling Zhang, Mingyi He, and Yingli Tian. Self-supervised modal and view invariant feature learning. arXiv preprint or 2rx::2005.11 1416,, 2020.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017. 7
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14x::1112.6980 2014. 25
A Sophia Koepke, Olivia Wiles, Yael Moses, and Andrew Zisserman. Sight to sound: An end-to-end approach for visual piano transcription. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1838 1842. IEEE, 2020. 4
Adarsh Kowdle, Christoph Rhemann, Scan Fanello, Andrea Tagliasacchi, Jonathan Taylor, Philip Davidson, Mingsong Dou, Kaiwen Guo, Cem Keskin, Sameh Khamis, et al. The need 4 speed in real-time dense visual tracking, ACM Transactions Graphics (TOG), 37(6):1-14, 2018. 2
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25:1097-1105, 2012. 2
Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, and Caroline Pantofaru. Virtual multi-view fusion for 3d semantic segmentation. Tn European Conference on Computer Vision, pp. 518-535. Springer, 2020. 2, 4
Fahad Lateef and Yassine Ruichek. Survey on semantic segmentation using deep learning techniques.
Neurocomputing, 338:321-348, 2019. 2
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: Bundle Aaidle-Adjustng Neural Radiance Fields. In ICCV, October 2021. URL https://arxis: wttpaaari.oorra////////////// 06405. 4, 5
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Europeon conference on computer vision, pp. 740-755, 750.755.66 Springer, 2014. 2
Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Ficguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey. International journal of computer vision, 128(2):261-33)) -18, 2020a. 2
Lingje Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Sua, and Christian Theobalt. Neural Sparse Voxel Fields. In Adv. Newral Inform. Process. Syst., 2020b. URL https://arxiv.. org/ ab00 11571. 3, 6
Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang, Xuan Wang, Jue Wang, and Pedro Sander. Deblur-nerf: Neural radiance fields from blurry images. arXiv preprint arX4::2111.14292, 2021. 3, 18
Ricardo Martin-Brusst
Noha Radwan, Mehdi Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Noural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021. URL httpa://arziv..ott
02268. 4
Ruben Mascaro, Lucas Teixcira, and Margarita Chli. Diffuser: Multi-view 2d-to-3d label diffusion for semantic scene segmentation. In IEEE International Conference on Robotics and Automation (ICRA 2021) (virtual), 2021.
21

---

Published in Transactions on Machine Learning Research (July/022)
Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEFE international conference on compuler vision workshops, pp. 37 45, 2015.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, ((2):99-100) 1995. 5
John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In 2017 IEEE International Conference on Robotics and automation (ICRA), pp. 4628 4635. IEEE, 2017.
Quan Meng. GNeRF: GAN-Based Neural Radiance Field Without Posed Camera. In ICCV, October 2021.
URL httpp::/aarx...ogg/bbb/////// 15606. 4
Lars Meschoder, Michael Occhsle, Michael Niemeyer, Scbastian Nowozin, and Andreas Geiger. Occupancy Networks: Learning 3D Reconstruction in Funetion Space. In CVPR, 2019. URL https://arxiv.orr
hbs/1812.0388B, 4,
Ben Mildenhall, Pratul Srinivasan, Matthew Taneik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, pp. 405 421. Springer, 2020. URL
orggabs/2000.. 08934. 3, 4, 5, 6, 8, 9, 25
Shervin Minace, Yuri Y Boykov, Fatih Porikli, Antonio J Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. 3
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv:220.005089, January 2022. 5, 18
Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-end 3d scene reconstruction from posed images. In ECCV, 2020. 2,
Arsha Nagrani, Chen Sun, David Ross, Rahul Sukthankar, Cordelia Schmid, and Andrew Zisserman. Speech2action: Cross-modal supervision for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Michael Niemeyer and Andreas Geiger. GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. In CVPR, 2021. URL https://arxiv wtt.://ogg/aaa/aaaaeaea //tp//01....11... 12100.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. In CVPR, 2020. URL attps://arxiv org/abs/19/t http://912.0001 07372.
Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-Vis View Reconstruction. In ICCV, October 2021. URL petps://arxia org/abs/
10078. 4
Jeong Joon Park, Pete Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In CVPR, 2019. URL https: //arxiv. http:/orr//abb//...00////.
4
Sida Peng. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. Tn CVPR, 2021. URL https://arxiv. wwtpsiiiirrg///as.e. org/abs/. 4
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision (ECCV), Cham, August 2020. Springer International Publishing.
22

---

Published in Transactions on Machine Learning Research (July/2022)
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pollern recognition, pp. 652 660, 2017a. 2, 6,
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning point sets in metric space. In Advances in neural information processing systems, pp. 5099-5108, 2017b. 4,
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. DeRF: Decomposed Radiance Fields. https://urziv.ory/ wtt.arrww....gvaaaeee
www./agg////eeeeeeoeeeeeee///w a01/2201.11/ 100,4000,, 2020.
Konstantinos Rematas, Ricardo Martin-Brualla, and Vittorio Ferrari. ShaRF: Shape-conditioned Radiance 08860. 4
Fields from a Single View. In ICML, 2021. CRL
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisled intervention, pp. 234-241. Springer, 2015. 2
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: Generative Radiance Fields for Aware Image Synthesis. Tn Adu. Neural Inform. Process. Syst., 2020. URL https://aaat /ttp::///axvvvvvo wwwriiv.org... org..oggg/oo.... org/abs/ /bbs/ 2007. 02442. 4
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint Xrxi::1906. 05797, 2019.
Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison. iMAP: Implicit Mapping and Positioning in Real-Time. In TCCV, October 2021. URL
abt///as//1000 12352. 4
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 2446 2454, 2020.
Towaki Takikawa. Neural geometric level detail: Real-time rendering with implicit 3d shapes. In CVPR, 2021. URL /ttps://arxiv.... /ttaa////.......eaaaeeete...t.ea 10994. 4
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-NeRF: Scalable large scene neural view synthesis. arviv, 2022. 18
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on compuler vision, pp. 4489 4497, 2015. 2
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. arxiv, 2021. URL https: //arxiv. wtt.///gg/////.....//////// 03907. 29
Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEFE/CVF Conference on Computer Vision and Patlern Recognilion, pp- 4604-4612, 2020. 4
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. IBRNct: Learning Multi-View Image- Based Rendering. In CVPR, 2021a. URL https://arxii httppp://aaaxv..o//abb////////// org/abs/ /t.e.aa.bb/2022.300... 13090. 3,
Zirui Wang, Shangzhe Wu, Weidi Xic, Min Chen, and Victor Adrian Prisacariu. NeRF-; Neural Radiance Fields Without Known Camera Parameters. htttpss//arriiu.mm///////2222000 2021b. 5
23

---

Published in Transactions on Machine Learning Research (July /
Zifeng Wu, Chunhua Shen, and Anton Van Den Hengel. Wider deeper: Revisiting the resnet model for visual recognition. Poltern Recognition, 90:119-136 2019. 9, 25
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond, 2021. URL https:/mmouralfielldss
bro.n. 2
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronan Basri, and Yaron Lipman. Multiview Neural Surface Reconstruction by Disentangling Goometry and Appearance. In Adv. Neural Inform. Process. Syst., 2020. URL https://arxis: http://axxii.oraaa//////a// wtwp:/as///........///// 09852.
Lin Yen-Chen, Pele Florence, Jonathan Barron, Alberto Rodriguez, Phillip Isola, and Tsung-Y Yi Lin. iNeRF: Inverting Neural Radiance Fields for Pose Estimation. In IROS, 2021. URL sstps://arxia org/abs/ abs/ 2012.05877. 5
Alex Yu, Sara Fridovich-Keil Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint wrXie:2112.05133 2021a.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for Real-Time Rendering of Neural Radiance Fields. In ICCV, October 2021b. URL http::tt/////xii.rrrrr http://wwwwoor/a////////// 14024.
31
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural Radiance Fields from One 02190.
Few Images. In CVPR, 2021c. URL
Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Rendering. In CVPR, 2021. URL https:
warrxiv.org/bb//1101..1 01202. 4
Greg Zaal, Rob Tuytel, Rico Cilliers, James Ray Cock, Andreas Mischok, Sergej Majboroda, Dimitrios Savva, and Jurita Burger. Hdri haven. hhttps: httpee///tt:///oll//// httpl//lllll..n.mm/o/ee com/hdris, 2021.
Cheng Zhang, Zhi Liu, Guangwen Liu, and Dandan Huang. Large-scale 3d semantic mapping using monocular vision. In 2019 IEEE 4th International Conference on Image, Vision and Compuling (ICIVC), pp. 71-76. IEEE, 2019. 4
Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radiance fields for largo-scale scene reconstruction. CVPR, 2022. 18
Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. arXiv preprinl XrXi::2106.0199 01977, 2021.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-Place Scene Labelling and Understanding with Implicit Scene Representation. In ICCV, October 2021. URL attps://arxiv.orr abs/2103. 15875. 3, 4, 6, 9, 10, 25
24

---

Published in Transactions on Machine Learning Research (July/2022)
NeSF: Neural Semantic Fields
for Generalizable Semantic Segmentation of 3D Scenes
Appendix
A Training Details
We describe the model architecture and training procedure used by NeSF and its baselines below. Unless otherwise stated, we train each method using all TRAIN SCENES with 9 randomly-selected images per scene. For KLEVR, this results in 100 TRAIN SCENES; for ToyBox5 and ToyBox13 500 TRAIN SCENES cach. All methods are evaluated on 4 randomly-selected images from each dataset's NOVEL SCENES. For KLEVR, this results in 20 NOVEL SCENES; for ToyBox5 and ToyBox13, 25 NOVEL SCENES each. We ensure that each method observes the same randomly-selected set of images per scene by specifying the seed of the random number generator.
NeRF. The first stage of NeSF is the training of per-scene NeRF models. We employ the model architecture and training regime of Mildenhall et al. (2020). Each seene's density field is described by an MLP with 8 hidden layers of 256 units, and its appearance by an additional MLP of 1 hidden layer and 128 units. We employ 7 octaves for positional encoding. Fach NeRF model trained on pixels selected at random from 9 views with the Adam optimizer (Kingma & Ba, 2014). The learning rate is exponentially decayed from 1e-3 to 5.4e-4 over 25,000 steps. We train each NeRF model for approximately 20 minutes on 8 TPUv2 cores.
NeSF. NeSF has two major model components: a 3D UNet and an MLP Decoder. For the 3D UNet, we employ the UNet architecture of Çięek et al. (2016) with the BatchNorm layers removed and only 2 max-pooling operations. We use 32, 64, and 128 output channels prior to cach max-pooling operation. For the MLP Decoder, we employ 2 hidden layers of 128 hidden units each with ReLU non-linearily.
We train NeSF with Adam optimizer (Kingma & Ba, 2014). We use exponentially decaying learning rate initialized to 1e-3 and decaying to le-5 over 25,000 steps. At cach step, we employ a stratified sampling approach: we randomly select 32 scenes, then randomly select set of 128 pixels from each scene's 's TRAIN CAMERAS. For volumetric rendering, we sample 192 points along each ray according to the stratified approach employed in NeRF (Mildenhall et al., 2020). For each scene in the batch, we discretize NeRF's density grid by probing at 64 evenly-spaced points. Before discretizing, we apply a random rotation about the z-axis (upwards) to each scene. For smoothness regularization, we uniformly sample 8,192 additional 3D coordinates from each scene and add random noise with standard deviation 0.05. When computing the loss, we assign weight of 0.1 to the smoothness regularization term
We find that we are able to train NeSF to convergence in approsimately 45 minutes on 32 TPUv3 cores.
Semantic-NeRF. We implement Semantic-NeRt following the architecture design of Zhi al. (2021). The architecture follows the same design as the NeRF architecture previously described with the addition of an MLP with 2 hidden layers of 128 units for modeling the semantics. We use 7 octaves for positional encoding and follow the stratified sampling approach employed in NeRF (Mildenhall et al., 2020) with a total of 192 samples (64 coarse and 128 fine). We train with Adam optimizer (Kingma & Ba, 2014), using an initial learning rate of 1c-3 that is decayed exponentially to 5.4c-4 over the course of 25,000 steps. We train a single Semantic-NeRF model per-scene for the NOVEL SCENES. Each Semantic-NeRF model trains for approximately 20 minutes on TPUv2 cores.
DeepLab. We train DeepLab Wide ResNet-88 model (Wu et al., 2019), warm starting with checkpoint pre-trained on COCO. For our optimization scheme, we apply SGD + Momentum with slow start lcarning rate of le-4 and a linear ramp up to 6e-3 followed by a cosine schedule decay in learning rate to 1.26e-7 at 55,000 training steps. We additionally apply weight decay of 1.0e-4. For each train step, we use a batch size of 32. Models are trained on 32 TPUv3 chips. To enable re-use of a well-performing hyperparameter configuration, we up-sample our input images from 256x256 to 1024x1024, using bilinear interpolation for the RGB input and nearest neighbor interpolation for the corresponding semantic maps.
25

---

Published in Transactions on Machine Learning Research (July/2022)
125
Degth
Zeepl
Spenetoewiel
Label
Figure 9: Multiview Consistency While NeSF and SparseConvNet classify the orange couch and the display identically from multiple independent views of the same scene, DecpLab's prodictions vary.
SparseConvNet. Our SparseConvNet (Graham et al., 2018) implementation is based on the TF3D (Huang et al., 2020) and 2D3DNet (Genova et al., 2021) implementations. Each convolutional layer except the last is an occupancy-normalized 3x3x3 sparse spatial convolution followed by batch norm and then ReLU. The final layer omits batch norm and ReLU. Each encoder stage is a pair of convolution layers followed by a 2x2x2 spatial max-pool operation, and each decoder layer is a voxel unpooling operation followed by a pair of convolutional layers. The encoder feature widths are (64, 64), (64, 96), (96, 128), (128, 160), (160, 192), (192, 224), (224, 256). These are the output channol counts of the first and sccond convolutional layers per block. The bottleneck is sequence of two convolutional layers of widths 256 each. The decoder feature widths are (256, 256), (224, 224), (192, 192), (160, 160), (128, 128), (96, 96), (64, 64). Finally apply a sequence of three convolutional layers with sizes (64, 64, class count), which are followed by softmax layer and a cross-entropy loss function. Our imput features are only occupancy (i.e., 1 on all input points). We use 0.005-width voxels a [-1, 1] cube scene. We optimize for 450,000 steps with SGD using a momentum of 0.9, a batch size of 5, an initial learning rate of 0.015, and a cosine learning rate decay starting at step 200,000 and ending at step 450,000. We add an l weight decay loss of le-4 and train asynchronously on 20 NVIDIA V100 GPUs. We apply the following data augmentations: XY rotations of up to =10 degrees, z rotations of 1800 degrees, and random scale factor between 0.9 and 1.1.
B Analysis
B.1 Qualitative Results
In Figures 14, 15, and 16, we present randomly-selected qualitative results on each dataset studied in this paper. In each row, we depict the ground truth RGB, depth, and semantic map alongside 2D segmentation maps produced by NeSF, DeepLab, and SparseConvNet. We observe that all methods are effective at separaling foreground objects from the floor and background. Unlike SparseConvNee, NeSF and DeepLab tend to assign different parts of the same object to different semantic categories when the correct category is ambiguous.
While NeSF and SparseConvNett are multi-view consistent by design, this is not the case for 2D methods such as DeepLab. In Figure 9, we demonstrate one instance of 3D inconsistency. In this example, NeSF and SparseConvNet label the orange couch and white-blue display identically from both views, whereas DeepLab's classification changes.
NeSF 3D Density Field Quality. Notably, a key difference between SparseConvNel and NeSF is the provision of a ground truth point cloud as input for SparseConvNet, Several aspects of SparseConvNet may contribute overall superior performance relative to NeSF including access oracle 3D gcometry, sparse point cloud input representation, or the SparseConvNet model architecture. To better understand where headroom exists for improvement of NeSF, begin by visually inspecting the difference in 3D geometry
26

---

Published in Transactions on Machine Learning Research (July/202]
KLEVR
TOYBOX-5
TOYBOX-13
NoSF Density Field
GT Point Cloud
Figure 10: NeRF 3D Density Field (top) and Ground Truth Point Clouds (bottom) for KLEVR, ToyBox5, and ToyBox13. We find that NeRF's density field accurately captures the 3D geometry of the scene. The NeSF density fields are sampled at resolution of 128x128x 128x128, and are filtered for ease of visualization for positive values with thresholds of 16, 64, and 64 for the 3 datasets respectively.
hyper- parameters Data augmentation, in the form of random scene rotations, increased spatial resolution of the donsity grid, and increased UNet model capacity improve 2D and 3D mToU. Experiments on 500 scenes from the ToyBox5 datasct.
Table 7:
Ablation:
Hyperparameter
2D
3D
Random Rotations
No
69.5
83.6
Yes
78.8
89.7
(32, 32, 32)
71.1
81.5
Density Grid
(48, 48, 48)
76.4
89.3
(64, 64, 64)
78.8
89.7
(16, 32, 64)
80.6
89.1
UNet
(24, 48, 96)
80.1
89.8
(32, 64, 128)
79.0
89.8
(0, 32)
78.6
90.7
(1, 32)
79.7
89.8
MLP
(1, 64)
80.7
89.4
(2, 128)
79.2
89.5
between the 3D density field of NeSF in relation the ground truth point cloud provided SparseConvNet for scenes selected from KLEVR, ToyBox5, and ToyBox13 in Figure 10. We observe the NeSF density fields often miss thin structures and fine details, and "floaters" are particularly evident in ToyBox13. Improvement of density field quality via improvements to NeRF representations may resolve such artifacts. Furthermore, akin to results in Table 8, such improvements would likely improve the performance NeSF. We leave additional inspection of 3D input representation and replacement the semantic model architecture of NeSF for future work.
B.2 Ablations
Model Ablations - Table 7. We repeat our our ablation study on the ToyBox5 model and observe overall consistent results with KLEVR model ablations. Table 7 shows results varying each component. Similar to the results on KLEVR, we observe that data augmentation in the form of random scene rotations improves quality the most, adding 9.3% and 6.1% to 2D and 3D mIoU respectively. The spatial resolution of the probed NeRF density grids is again confirmed as crucial, and notably to greater extent than for XLEVR. We hypothesize that this occurs ToyBor5 contains more fine structured objects than KLEVR.
27

---

Published in Transactions on Machine Learning Research (July/2022
Table Ablation: sensitivity to recon- struction quality for ToyBox5 The accuracy of method improves with NeRF's reconstruc- tion quality. PSNR and SSIM are averaged across all scenes and metrics aggregated. Exper- iments on all scenes from ToyBox5.
RGB
NeRF
NeSF
Images
PSNR
SSIM
2D
3D
5
17.5
15.0
17.9
10
2.9
0.15
29.1
35.2
25
23.9
0.76
61.4
74.1
50
26.3
0.81
72.3
88.7
75
27.3 2.0
0.83 1 0.05
72.6
89.5
100
27.9 2.0
0.84 0.8410.00
73.6
90.0
100
2D mtou
mloU
60
85
40
50 100
175
250
375
500
Scenes
Ablation: data efficiency 2D and 3D mloU as a function the number train scenes for scenes with supervision from 1, 2, 5, 10, 25, or 50 semantic maps per scene. Note, 50 semantic maps did not fit in memory for 500 scenes, hence this particular setup is dropped from the results. NeSF generalizes to scenes with few improve the accuracy. Experiments on ToyBox5 dataset.
Figure 11:
semantic map per scene. Additional semantic maps per scene marginally
Sensitivity to reconstruction quality - Table 8. We ro-evaluate the robustness NeSF to NeRF reconstruction quality the context the ToyBox5 dataset. To modulate reconstruction quality, we vary the number of RGB images used when fitting NeRF models from to 100 and confirm NeRF reconstruction quality improves more RGB images provided. As previously observed, the 2D and 3D segmentation quality of NeSF improves monotonically with NeRF's reconstruction quality, Notably, 3D segmentation accuracy begins to level off near 88% when NeRF models are optimized with as few 50 RGB images, with large jump in performance between 25 and 50 images.
Sensitivity to data scarcity Figure 11 We repeat our analysis providing limited numbers semanti- cally labelled maps for NeSF's training on the ToyBox5 model. We vary the number provided label maps from to 50. Similar the KLEVR setup, we observe that providing additional semantic maps per scene improves the performance, with large jump between 5 and 10 maps. A saturation in the model performance is reached at around 25 maps per scene. Moreover, the model is still able to generalize with as little as semantic map per scene.
Comparison to NeRF SparseConvNet Table 9 To better understand the performance gap between SparseConvNet and NeRF, we replace SparseConvNet's oracle point cloud with one derived from NeRF's density field. In particular, we derive point clouds from NeRF by sampling the density field on grid, thresholding to only include points above heuristically chosen density value (a 64), and removing all
Table 9:
SparseConvNet Ablation
Point Cloud
3D mloU
NcSF compared to SparscConvNet modcls trained on NeRF-derrved and oracle point clouds the ToyBox13 dataset.
NeSF
0.62
SparseConvNet SparseConvNet
NeRF
0.68
Oracle
0.82
28

---

Published in Transactions on Machine Learning Research (July/2022)
ToyBox5
ToyBox13
TRAIN CAMERAS
TEST CAMERAS
TRAIN CAMERAS
TEST CAMERAS
2D mloU
3D mloU
2D mloU
3D mloU
2D mloU
3D mloU
2D mloU
3D mloU
NeSF NeSF(+rgb features)
79.5 1.3
89.1 001100. 0.9
78.8 1.4
90.2 = 0.8
54.3 2.0
58.7 = 1.7
54.0 2.2
60.0 2.6 60.1 + 1.3
80.0 1.3
88.8 + 0.8
79.8 + 0.8
89.1 + 1.1
54.7 0.9
59.2 0.9
54.2 + 1.5
Table 10: NeSF with RGB Features NeSF with RGB Features performance relative to NcSF. We extend geometric density features to include RGB features sampled at an arbitrary but consistent viewing direction. The features are attenuated by the respective positional density of the field at their location in the grid, and NeSF is trained with the extended set of features. We do not observe a significant difference in performance given the particular datasets used for evaluation.
RGB Reference
Semantic Reference
EPI
Depth
Epipolar Plane We demonstrate the 3D consistency of NeSF by rendering the epipolar plan along the red scan line as the camera moves from right to left. The epipolar plane is is smooth and consistent. except when a "floater" passes in front of the camera.
Figure 12:
points within the dense floor below a cutoff height (z=-0.55). Semantic labels are provided by transferring from the nearest neighbor point in the respective oracle point cloud. The labeled point clouds are used train SparseConvNet model, and inference is performed unlabelled NeRF point clouds for test scenes. For evaluation, mloU is calculated by comparing oracle point cloud labels against the nearest neighbor label per point from the predicted labels for NeRF point cloud test scenes. Though still a slightly unfair comparison due to the use of ground-truth truth 3D data for the purpose of labeling the NeRF point clouds, we find the experiment informative; demonstrating the far smaller gap between NcRF-SparseConv and NeSF performance. The smaller performance gap indicates the init significant, performance difference may be largely attributed to imperfect geometric reconstructions, ill-defined density within objects, and/or floater artifacts resulting from NeRF based geometry reconstruction. Due to extra time and resource overhead for training SparseConvNet,, we limit the experiment to most challenging dataset ToyBox13.
We explore the utility of extending the density field input to the UNet of NeSF to additionally contain the RGB features from the radiance field of the NeRF. To sample RGB features along a spatial 3D grid, viewing directions along with positions must be provided to the NeRF MLP. While possible to sample radiance along a calculated surface normal, NeRF geometry is known to be foggy as opposed to being tightly concentrated at surfaces, making normal vectors quite noisy when calculated (Verbin et al., 2021); moreover, such normals are meaningless in empty space well as within objects. As an alternative for our datasets specifically, we consider that the ShapeNet objects utilized for the ToyBor5 and ToyBox13 datascts are composed of diffuse materials, and radiance can be considered independent of vicwing direction. As such, when sampling the radiance feat ures along the 3D spatial grid, we simply sample at an arbitrary viewing direction (i.e. (1,1,1)). To ensure features are representative of filled space, we attenuate the RGB fcatures by the density field, and concatenate the density as well. In our experiments we observe performance remains essentially on par with the use of density alone, but suspect results may be specifically due to lack of strong correlation between object bolorttexture and class in our datasets.
Addition of RGB features Table 10
29

---

Published in Transactions on Machine Learning Research (222)
Figure 13: Floaters In the above 3 rows, we illustrate NeRF's RGB reconstruction (top), NeSF's semantic field, and NeRF's s density field (bottom) over the course of 5 video frames. When NeRF's density field contains "floaters", NeSF often assigns them to the background semantic category. Note that recent methods such as Mip-NeRF-360 (Barron et al., 2022) have recently introduced additional losses to remove such artifacts.
B.3 Multiview Consistency
Unlike conventional 2D methods, NeSF is 3D-consistent by design. In Figure 12, we visualize the epipolar plane traced out along the red reference line for NeSF's semantic predictions. We find that the resulting predictions are consistent and smooth except when a floater obstructs the camera's view, as illustrated by the white smudge in the EPI. In Figure 13, we see further example of this phenomena. As the camera rotates about the scene, floating mass of density obstructs the camera's view, and the resulting semantic maps contain large number of mislabeled pixels. We strongly encourage the reader to view additional results in the accompanying video for more detail. In particular, results depicting floaters for ToyBox5 can be viewed from timestamp 7:34.
B.4 Limitations
Confusion matrix. In Table 11 and Table 12, present NeSF's per-class confusion matrix the ToyBox13 dataset for 2D pixel and 3D point classification. While NeSF is able to easily identify larger, articulated semantic categorics such as cabinet, chair, display, or table (78.0-8..4% 2D, 79.4-93.6% 3D), struggles to identify object categories for smaller objects such as rifle (56.3% 2D, 75.3% 3D) or geometrically unarticulated objects such as loudspeaker (38.5% 2D, 40.4% 3D). When NeSF confuses foreground object categories, the most common errors are between geometrically-simiil classes. For example, benches often mislabeled chairs (17.0% 2D, 17.9% 3D) and sofas (26.5% 2D, 27.4% 3D), and loudspeakers are often mislabeled tables (32.0% 2D, 32.5% 3D).
Accuracy 2D vs. 3D. Our experiments indicate that NeSF's accuracy is higher in 3D than in 2D. We found this surprising, especially considering the 2D nature of NeSF's semantic supervision. We believe the ultimate cause to be "floaters" in the 3D density field recovered by NeRF. In Table 11, we see that approximately 5-10% of 2D pixels from each semantic category are mislabeled "background". In contrast, Table 12
30

---

Published in Transactions on Machine Learning Research (July/2022)
demonstrates that the same type of error is made approximately 1% of the time in 3D. The most prominent exception to this is the bench category, whose objects often contain thin structures poorly captured by NeSF.
Impact of floaters. In Figure 13, we qualitatively show how "floaters" reduce NeSF's accuracy in image- space. In this of video frames, we demonstrate a camera path passing front of a floating cloud. This cloud is assigned to the background semantic category and obscures the foreground objects from the scene. In spite of the objects being correctly labeled, the generated semantic maps are incorrect. As result, NeSF achieves lower 2D mIoU than 3D mloU the latter is not hindered by floaters and is corroborated by Tables 11 and 12. We believe that climinating this failure in geometric construction will significantly improve NeSF's accuracy. Solutions are readily provided by methods building on NeRP (Yu et al., 2021b). We strongly encourage the reader to view additional results in the accompanying video for more detail.
baarlegronnd
steplann
beinch
oshinct
car
chair
dicolay
b-mp
Tentlppahhee
endo
Tablin
telophhion
serceal
Tarllgrond
0.0%
03075
0.07
0-375
3-175
0:13
0.075
0.103
0.0%
0.27
0.03
0.07
beach
0:155
0.7%
42.15
0.23)
0,05
17.00
0.09
0.27
3.0%
1.23%
20.0%
1.79%
0.07
1.4%
cabinet
2-2%
0.038
0.05
78.08
0.055
0.198
2.7%%
0.09%
12.55
0.09
0.0%
2.79
1.8%
0.0%
thair
3.8%
0.0%
0.9%
0.0%
0.055
$9.4%
0.19)
0.0%
0.15
0.0%
2.9%
0.7%
0.0%
0,0%
display
5.2%
0-03
0.3%
0-4%
0.155
1.2%
68.350
0.0%
5.7%%
0.0%
6.2%
1.59
0.2%
0.0%
londapiake
3.4%
0.15
4.3%
10.956
0.05
6.09
5.29
1.1%
88.8%
0.035
2.2%
030
1.7%
0.4%
rife
30.756
1-638
2.0%
0.2%
0.055
0.79
5.19%
0.0%
3.00%
$6.356
3.0%
0.795
0.0%
1.1%
table
5.475
5.125
0.035
celephons
1.75
0.03
0.0%
9.39
0.05
0.29
5.79%
0.09
15.79
0.09
6.0%
0.595
09.05%
0.0%
0.055
4:335
Table 11: Confusion matrix for 2D semantic segmentations by NeSF on ToyBox13. Each row
corresponds to a ground truth label and is normalized to sum to 100%. NeSF's most common errors include confusing similarly-shaped objects and classifying small and thin objects as background. Correct classifications are highlighted in bold.
Tackgroond
strplaue
betcch
cablant
chair
dicalay
bump
Teudeppakee
ridfe
sofs
Eelophaiin
Aussell
sirplane
78.07t
3.6%
0.07
0.07
3.395
0.5%
3.375
2.1.%
0.0%
0.1%
7.6%
beach
0.000
0:811
82:75%
0-25
0.05
17.096
0.29%
0.35%
9.10%
1.7%%
27.43
1.550
0.05
1.4%
0.8%
0.055
86.7%%
0.07
5.375
0.2%
5.45
0.3%
9.1%
4.3%
thatz
1.000
0.036
1.1%
0.05
0.05
95.63
0.19
0-1%
9-19
0-136
8.1%
0.590
0.0%
0.0%
lainp
0.37
0.15
00.7%
20
0.05
6.87
0.9%
londsprake
1.1%
0.158
4.4%
12.356
0.15
0.4%
5.09
1.2%
40.45%
0.050
2.1%
52 850
1.0%
0.4%
sofs
6.75
0.05
0.075
22.078
3.395
0.1%
0.0%
68.275
0.075
0.075
table
0.65
0.036
1.8%
5.396
0.05
4.7%
0.09
5.29
1.85
0.196
0.8%
$2.836
0.0%)
0.8%
relephons
0.05E
19.59
89.15
1.158
0.0%
0.075
0.016
0.395
0.0%
80.7'
Table 12: Confusion matrix for 3D semantic segmentations by NeSF on ToyBox13. Each row
corresponds to a ground truth label and is normalized to sum 100%. NeSF's most common errors include confusing similarly-shaped objects and classifying small and thin objects as background. Correct classifications are highlighted in bold.
31

---

Published in Transactions on Machine Learning Research (July/2022)
QE 1e52
Dewplab
Figure 14: Additional Qualitative Results on KLEVR
32

---

Published in Transactions on Machine Learning Research (July/2022)
Degpcab
Figure 15: Additional Qualitative Results on ToyBox5
33

---

Published in Transactions on Machine Learning Research [July/2022]
Degpcab
Figure 16: Additional Qualitative Results on ToyBox13
34


Reviews:
[{"review_id": "e37RXbtuga", "reviewer": "Reviewer_KQCV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros: \n- Novel and technically sound approach. \n- Writing is mostly clear and easy to parse. The structure of the paper is well organized, and the figures are clear and visually appealing.  \n- The semantic component can generalize to novel scenes.\n- The geometry-only method demonstrates robustness to color shifts compared against the baseline etc. \n- Thorough literature review and well-written limitation sections. \n\nCons:\n- The method has several drawbacks compared to its closest related work, Semantic-Nerf (geometry only, modular pipeline, redundant representation). \n- Experiments are only conducted in synthetic datasets. Overall it's hard to see the practical values. \n- Both 2D and 3D baselines are not up-to-date. Semantic-Nerf used a weak segmentation network. \n- The merits of the dataset are questionable. \n- There are a few ambiguous claims on existing works and datasets. \n\nThe proposed geometry-only semantic field has a few drawbacks: 1. it requires two different representations at inference (one is Nerf representation, and the other is 3D voxel). 2. Error could be propagated along a modular pipeline — i.e., incorrect geometry from Nerf (likely to be noisy in practice as demonstrated in prior literature) will result in inaccurate 2D/3D semantics. 3. Appearance information is not well exploited.  \n\nThe limitation section is well-written. That said, the submission would be significantly stronger if the author could at least resolve one or few of the discussed limitations, e.g., demonstrate real-world images or leverage appearance information provided by the neural radiance field. \n\nI have some reservations about the merits of the proposed dataset.\nThis synthetic dataset is built upon basic object shapes and geometry primitives, and procedurally-generated object placements. It's hard to say how much benefit it could bring for advancing vision and graphics in practice. As a comparison, photorealistic rendering datasets, such as HyperSim and Synscapes, provide sensor photorealism while the underlying layout and assets are much more realistic than the proposed dataset. In addition, PBR rendering tools like BlenderProc and Clevr, provide users a reproducible script interface to generate a physical-based rendering dataset with similar quality but much better flexibility and controllability. The proposed dataset, however, is fixed. \n\nBoth DeepLab and SparseConvNet are quite old semantic segmentation networks published four years ago. It is highly recommended to use stronger 2D/3D baselines and a stronger 2D net for Semantic-Nerf. \n\nDespite that robustness to color dataset shift demonstrated, relying on geometry makes it hard for NeSF to handle certain situations, e.g., semantically distinguishing a volleyball vs. a basketball or wall vs. painting. \n\n\nClaims:\n- \"no RGB input is required for inference on novel scenes\" this sentence could be misleading, considering building nerf requires RGB input. \n- \"Our method is the first to learn semantics from the geometry stored within a 3D neural field representation.\" I would be careful about this claim as it could be misleading. As the author mentioned, The pseudo-labeled version of Semantic-Nerf requires only posed RGB images as well. The semantics and geometry for semantic-Nerf are jointly learned under a unified representation.  \n- \"they lack the scale, detail, and precision necessary to simultaneously evaluate 2D and 3D semantic segmentation\": I have some reservations about the claims on existing datasets. ScanNet and Matterport3D have been widely adopted and used in common benchmarks. Both are large, captured in the real world, and can provide accurate 2D and 3D semantic benchmarks. \n- \"NeSF is trained using purely 2D signals, and can be trained with as few as one labeled image per scene\" — this could be the drawback since semantic nerf only requires 2D annotated images without building a nerf during training. Training images could be single-shot instead of a multi-view collection. \n\nQuestions: \n- Have you tried to leverage the radiance field into the UNet input as well? This should be fairly straightforward.  \n- How about training SparseConvNet on the 3D voxelized occupancy field extracted from Nerf or dense point cloud produced by a SFM-MVS pipeline. Either way could provide a fair comparison. \n- What will be the downstream applications? Do you anticipate such a method could replace semantic fusion for visual scenes in the future? \n- Do you anticipate the method will work well for real-world datasets such as ScanNet?\n\nMinor:\n\nAnother interesting work worth discussion (not required as it's public post the submission): \n[1] Fu et al. Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation, arXiv 2022. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This method proposed a novel semantic understanding method. The approach takes posed images as input and constructs a neural radiance field as an intermediate representation. The 3D translation UNet is applied to convert the opacity field to a semantic field, and a neural rendering algorithm produces a rendered semantic mask. The semantic components are generalizable in novel scenes. Experiments are demonstrated on a novel synthetic dataset. \n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See the \"Questions\" and \"Claims\" parts above. \n\nThe reviewer will also appreciate the author's response to some of the comments above. ", "broader_impact_concerns": "The paper has a well-written section discussing the potential broader impact concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MwZBf952jU", "reviewer": "Reviewer_kHpB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- While the framework is simple, enabling training with one labeled semantic map per scene shows good potential towards 3D understanding with more practical 2D supervisory signals.\n- I believe the main contribution of this paper is the use of neural fields for representing 3D data that can be derived from posed 2D images. This has the advantage of applying random geometric data augmentation that samples the density grid in the continuous space while training the 3D ConvNet. The 3D ConvNet and volume rendering (of semantic features) components are rather trivial.\n- I think the paper is well-written and the presentation is clear in general. The overall framework is easy to understand, and the experimental settings are described in a detailed manner with sufficient discussions.\n\nWeaknesses:\n- The new synthetic datasets are nice for benchmarking the performance of NeSF, but they are inherently a product from the Kubric paper, and I do not think they are considered as a contribution from this paper.\n- The comparison against SparseConvNet does not seem to be apple to apple, as the data modality is inherently different. The below baselines/ablations would make the comparison fairer:\n  - Use the ground-truth 3D shapes (which effectively has continuous occupancy) to train NeSF, or use the same pretrained NeRF representations, threshold the density to get a surface representation, then sample point clouds for SparseConvNet.\n  - Sample the point clouds dense enough to allow differentiable rendering, so that SparseConvNet can be learned from 2D supervision.\n  Otherwise, comparing against SparseConvNet under the current setting is pretty meaningless.\n- I think some discussions should be made on Fig 4, as it would be expected that mIoU should be strictly improving with the number of supervising semantic maps per scene.\n- I think there should at least be some discussions on the failure cases. Since NeSF depends heavily on the performance of NeRF, there should be some discussions on the robustness of NeSF to the quality of NeRF (e.g. when NeRF fails to optimize reasonable volume density fields). \n- Why is a decoder MLP necessary when one can directly predict and volume render the class label logits from 3D convolution? I think this should be justified. Also, it is unclear what the hyperparameters in Table 5 exactly correspond to.\n- Fig ?? is unreferenced in p12.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for 3D semantic segmentation without 3D supervision, specifically on the volume density of neural radiance fields (NeRF). Given a dataset of multi-view posed images that allows one to pre-convert to a dataset of NeRF representations, along with the corresponding labeled semantic maps for all or partial viewpoints, the authors advocate to train a 3D U-Net that takes a density grid sampled from the input density field to predict a 3D semantic grid. By volume rendering, the predicted 3D semantic grid can be converted to 2D semantic segmentation predictions. The authors also created 3 datasets with a new rendering engine with different scene complexities to evaluate the proposed method.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I would like to see all weaknesses fixed. Aside from the last weakness of referencing, I think the rest are all important to justify for acceptance.", "broader_impact_concerns": "The potential societal impact was sufficiently addressed in the broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r8WXTgE9xf", "reviewer": "Reviewer_u1Zd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I am positive about this paper. The paper has many strengths and some weaknesses that I point out below together with questions, comments, and suggestions for improvement.\n\n### **Strengths**\n\n- This paper takes a step forward in the direction of learning high-level semantics within neural fields -- this is an important and worthy problem to address.\n- The insight to operate on the density field is interesting and leads to modularity (NeRF can be pre-trained and distributed).\n- It uses only 2D semantic map supervision while lifting it to 3D. It could have broad applications since 2D labels are significantly easier to obtain than 3D labels.\n- The paper does a good job, especially in the introduction clearly stating the differences to previous work including NeRF and Semantic NeRF.\n- The paper presents a large synthetic dataset of over 1000 scenes.\n\n### **Weaknesses**\n\n- One of the main weaknesses of the paper is the use of synthetic data for experimentation. NeRF took us away from the clutches of synthetic data by showing results on real data. This paper takes us back to synthetic data. While I understand gathering real data at this scale is challenging, it would have also been more compelling -- especially because we have very good tools to extract semantic/instance segmentation masks on real data. Perhaps the authors could provide some more justification for the use of (only) synthetic data.\n\n- In Section 3, the paper mentions that it can be trained with unpaired segmentation maps. But this feature is never demonstrated. The ablations only evaluate scarcity of segmentation maps. It would make the paper stronger if this feature were to be demonstrated.\n\n- As pointed out in the limitations, NeSF relies solely on the density field which is only an indirect function of geometry and contains no information about color. On the other hand, 2D segmentation almost entirely depends only on the appearance. I hypothesize that the approach would work significantly better if appearance information is used.\n\n- The Data Augmentation section is lacking detail. It is unclear how much augmentation is performed and how this affects training times. Table 5 briefly discusses performance but more description would be nice to have.\n\n- In the synthetic data, are views sampled on a fixed radius hemisphere? If so, how does the method generalize to views of different radius?\n\n- It is unclear how the method performs when a new semantic class unseen at training is shown at inference time. What would happen when a new object never seen before appears at inference time?\n\n### **Other Questions/Comments/Suggestions** \n\n- The abstract mentions one-shot learning of semantics -- but it's unclear whether this is at inference time or training time (becomes clear later). Perhaps this could be made more explicit in the introduction?\n- How does the choice of epsilon in equation (4) affect the quality of results?\n- Page 12, line 4. Fix \"??\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents neural semantic fields -- a class of neural fields that encode scene semantics in addition to the radiance/shape. The method is trained on a collection of NeRF models (pre-)learned from sets of RGB images with corresponding semantic segmentation maps. At inference time, given a NeRF model of a new scene not seen before, the method is able to generalize to produce good semantic fields for new viewpoints. The central technical idea is to operate on the density field of NeRF using a 3D CNN to extract features that can estimate a semantic label. Detailed experiments show that the method outperforms other work on a large corpus of synthetic data.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\n- Due to my concerns about computational expense (see broader impact concerns), I would like to see a confirmation from the authors to release pre-trained NeRF models, code, and architecture.\n\nWith the above-requested changes, I would be in full support of acceptance.", "broader_impact_concerns": "The proposed method is computationally very heavy. Moreover, there are a lot of hyperparameters in the proposed work which could make replication hard. The paper addresses the compute question and proposes to distribute pre-trained NeRF models. I am satisfied that this addresses my replicability concerns.  ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After an extensive discussion between the reviewers and the authors, all three expert reviewers are in favor of acceptance. The editor agrees with the reviewers and is also in favor of acceptance. The discussion surfaced a few small changes that the reviewers and authors (at least in the editor's view) agreed upon.Here are the specific changes the editor sees as needed (and who asked for them):
1.  (kHpB, KCQV) Please include in the supplementary the SparseConvNet using noisy Nerf Geometry baseline discussed in kHpB Q2 (proposal #1 - the one that has already been completed; no need to do any new ones). This was also the one agreed upon with KCQV in Q3 .
2. (KQCV) Please address KQCV's Q2 request for a simple ablation (i.e., one that takes the second-to-last feature layer). 
3. (kHpB) Please add to Figure 4's caption to help readers understand it better. An abbreviated version of what was written in the response to kHpB on 17 May is perfect.
4. (u1Zd) Please add a few short sentences to address the discussion regarding u1Zd Q1 ("nerf took us away from the clutches of synthetic data"). The editor agrees with u1Zd that a simple demo could be nice, but also not necessary
5. (u1Zd) Please add commentary about data augmentation (Q4) and new semantic classes (Q6)

Based on the discussion between the reviewers and the authors, the editor has the following suggestions that are entirely up to the authors but may improve the final version of the paper:
1. The discussion with KQCV about the claim of the method replacing semantic fusion might be of interest to readers. If the authors can somehow integrate some of this discussion, it might be of interest to readers. If there is not a good spot, that is also fine.
2. A consolidated discussion of failure cases could be helpful. The paper does discuss failures throughout the paper. However, a short consolidated sentence or two flagged with "Failure cases" might help readers who are reading the paper more casually.

The editor also would like to thank the authors and reviewers for engaging in discussion productively throughout this process.
Decision: Accept (poster)