SegNeRF: 3D Part Segmentation with Neural Radiance Fields
Authors: Jesus Zarzar
Sara Rojas Martinez
Silvio Giancola
Bernard Ghanem
Venue: Submitted to ICLR 2023 (2023)

Abstract: Recent advances in Neural Radiance Fields (NeRF) boast impressive performances for generative tasks such as novel view synthesis and 3D reconstruction. Methods based on neural radiance fields are able to represent the 3D world implicitly by relying exclusively on posed images. Yet, they have seldom been explored in the realm of discriminative tasks such as 3D part segmentation. In this work, we attempt to bridge that gap by proposing SegNeRF: a neural field representation that integrates a semantic field along with the usual radiance field. SegNeRF inherits from previous works the ability to perform novel view synthesis and 3D reconstruction, and enables 3D part segmentation from a few images. Our extensive experiments on PartNet show that SegNeRF is capable of simultaneously predicting geometry, appearance, and semantic information from posed images, even for unseen objects. The predicted semantic fields allow SegNeRF to achieve an average mIoU of 30.30% for 2D novel view segmentation, and 37.46% for 3D part segmentation, boasting competitive performance against point-based methods by using only a few posed images. Additionally, SegNeRF is able to generate an explicit 3D model from a single image of an object taken in the wild, with its corresponding part segmentation.

Under review as a conference paper at ICLR 2023
SEGNERF: 3D PART SEGMENTATION WITH NEURAL RADIANCE FIELDS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in Neural Radiance Fields (NeRF) boast impressive perfor- mances for generative tasks such as novel view synthesis and 3D reconstruction. Methods based on neural radiance fields are able to represent the 3D world implic- itly by relying exclusively on posed images. Yet. they have seldom been explored in the realm of discriminative tasks such as 3D part segmentation. In this work, we attempt to bridge that gap by proposing SegNeRF: a neural field representa- tion that integrates semantic field along with the usual radiance field. SegNeRF' inherits from previous works the ability to perform novel view synthesis and 3D reconstruction, and enables 3D part segmentation from few images. Our tensive experiments on PartNet show that SegNeRF is capable of simultaneously predicting geometry, appearance, and semantic information from posed images, even for unseen objects. The predicted semantic fields allow SegNeRF to achieve average mloU of 30.30% for 2D novel view segmentation, and 37.46 % for 3D part segmentation, boasting competitive performance against point-based meth- ods by using only few posed images. Additionally, SegNeRF able to generate explicit 3D model from single image of an object taken in the wild, with its corresponding part segmentation.
y(x), d. Emii
3D Segmentation
Y(x), Emii
H
x(x). Emise
Source View
Encraeer
3D Reconstruction
0
Query Sources
2D RGB Target View d L Cyec
Volume
Besdering
2D Segmentation Target View
Ray Points Target. View d
Neural
Point Cloud
Point Grid
Reuderer
L L
Implicit Representation with Neural Radiance Fields for 2D novel view Semantic Segmentation, as well as 3D Segmentation and Reconstruction. Our model takes as input one or more source views of an object (top-left image). The source view is used to generate feature grid, which is queried with set of (i) ray points for volume rendering, (ii) object point cloud for 3D semantic part segmentation, or (iii) a point grid for 3D reconstruction. Training is supervised only through images in the form of 2D reconstruction and segmentation losses. However, at test time, our model is also capable of generating 3D semantic segmentation and reconstruction.
Figure 1: SegNeRF framework:
We will release our code publicly for reproducibility.

---

Under review as a conference paper at ICLR 2023
INTRODUCTION
Humans able perceive and understand the objects that surround them. Impressively, understand objects even though our visual system only perceives partial information through 2D projections, i.e. images. Despite possible occlusions, single image allows to infer an object's geometry, estimate its pose, and recognize its parts and their location in a 3D space. Inspired by this capacity of using 2D projections understand 3D objects, we aim to understand object part semantics in 3D space by solely relying on image-level supervision.
Most works learn object part segmentation by leveraging 3D supervision, ie. point clouds or meshes (Wang& & Lu, 2019; Qi et al., 2017b; Li et al., 2018). However, given the accessibility of camera sensors, is easier at inference time have access to a collection of images rather than a 3D scan of the object. To exploit real-world scenarios, it would thus be advantageous to have the ability of understanding 3D information from image-level supervision. For that purpose, Neural Radiance Fields (NeRF, Mildenhall et al., 2020) emerged as cornerstone work in novel-view syn- thesis. NeRFs showed how to learn a 3D object's s representation solely based on image supervision. While NeRF's seminal achievement was learning individual scenes with remarkable details, its im- pact grew beyond the single-scene setup, spurring follow-up works on speed (Müller et al., 2022: Yu et al., 2021a), scale (Martin-Brualla
al., 2021; Xiangli et al., 2021), and generalization (Jang & Agapito, 2021; Yu et al., 2021b). Despite the rapid advances in NeRF, only a handful of works (Zhi al., 2021; Vora al., 2021) have leveraged volume rendering for semantic segmentation or 3D reconstruction. Yer, these capabilities are a defining feature of 3D understanding, with potential applications in important downstream tasks.
In this work, we present SegNeRF:
an implicit representation framework for simultancously learn- ing geometric, visual, and semantic fields from posed images, as shown in Figure 1. At training time, SegNeRF only requires paired posed colored images and semantic masks of objects, inheriting NeRF's independence of direct geometric supervision. At inference time, SegNeRF projects appear- ance and semantic labels on multiple views, and can thus accommodate a variable number of input views without expensive on-the-fly fly optimization. SegNeRF inherits the ability to perform 3D recon- struction and novel view synthesis from previous works (PixeINeRF, Yu et al., 2021b). However, we demonstrate the ability to learn 3D semantic information through volume rendering by leveraging PartNet (Mo al., 2019), large-scale
semantic field. We validate SegNeRF's performance dataset of 3D objects annotated with semantic labels, and show that our approach performs on-par with paset-based methods without the need for any 3D supervision.
Contributions. We summarize our contributions as follows. (i) We present SegNeRF, a versatile 3D implicit representation that jointly learns appearance, geometry, and semanties from posed RGB images. (fi) We provide extensive experiments validating the capacity of SegNeRF for 3D part segmentation despite relying exclusively on image supervision during training. (iii) To the best of our knowledge, SegNeRF is the first multi-purpose implicit representation capable of jointly reconstructing and segmenting novel objects without expensive test-time optimization.
2 RELATED WORK
2.1 3D GEOMETRICAL REPRESENTATIONS
The classical representations for data-driven 3D learning systems can be divided into three groups: voxel-, point-, and mesh-based representations. Voxels are a simple 3D extension of pixel repre- sentations; however, their memory footprint grows cubically with resolution (Brock al., 2016; Gadelha et al., 2017). While point clouds are more memory-effieient, they need post-processing to recover missing connectivity information (Fan al., 2017: Achlioptas al., 2018). Most mesh- based representations do not require post-processing, but they are often based on deforming fixed size template mesh hindering the processing of arbitrary shapes (Kanazawa et al., 2018: Ranjan et al., 2018). To alleviate these problems, there has been strong focus on implicitly represent- ing 3D data via neural networks (Mescheder et al., 2019; Park et al., 2019; Sitzmann et al., 2019; Mildenhall et al., 2020). The fundamental idea behind these methods is using neural network (x) to model certain physical properties (e.g., occupancy, distance to the surface, color, density, or illumination) for all points in 3D space.
2

---

Under review as a conference paper at ICLR 2023
Most recent efforts have developed differentiable rendering functions that allow neural implicit shape representations to be optimized using only 2D images, removing the need for ground truth 3D shapes (Liu et al., 2019b; Niemeyer et al., 2020b; Yariv et al., 2020: Sitzmann et al., 2019). For instance, Sitzmann et al. (2019) propose Scene Representation Network (SRN), a recurrent neural network that marches along each ray decide where the surface is located while learning latent codes across scene instances. Neural Radiance Fields (NeRF, Mildenhall et al., 2020) encodes a scene with 5D radiance fields, which accurately rendered high-resolution RGB information per light ray. PixeINeRF (Yu et al., 2021b) extended NeRF to work on novel scenes and object cate- gories by using an image encoder condition a NeRF on image features to dispose of test-time optimization. In this paper, we treat PixeINeRF as a powerful generalizable 3D implicit represen- tation, and extend it to learn a semantic field along with the usual radiance field. This extension allows us to (1) predict 2D novel view semantics through volume rendering. the density and semantic fields with a bounded 3D volume or arbitrary 3D points to obtain both 3D object reconstruction and part segmentation.
(2) efficiently query
2.2 3D SEMANTIC SEGMENTATION
We discuss previous methods segmenting 3D data and focus on those aimed at 3D part segmentation.
Explicit Representations. Most existing works for 3D semantic segmentation are point-, voxel- and multi-view-baat based approaches. As for point-based methods, the seminal PointNet (Qi et al., 2017a) work excels at extracting global information, but its architecture limits its ability to encode local structures. Various studies addressed this issue (Qi et al., 201 Li et al., 2018), proposing different aggregation strategies of local features. Voxel-based methods achieve competitive results (Le & Duan, 2018; Song et al., 2017). However, they require high-resolution voxels with more detailed structure information for fine- fine-grained tasks like part segmentation, which leads to high computation costs. In contrast to these methods, we only use 2D supervision and inputs to segment dense 3D representation. Kalogerakis et al. (2017) acquire a set of images from several viewpoints that best cover the object surface and then predict and revise part labels independently using multi-view Fully Convolutional Networks and surface-based Conditional Random Fields. Nevertheless, it requires a ground truth 3D object to query points from at test time, while our method can reconstruct the object while segmenting it.
Implicit Representations. Similar our approach, Semantic NeRF (Zhi al., 2021), NeSF (Vora et al.. 2021), and Semantic SRN (Kohli et al., 2020) leverage implicit representations to predict se- mantic labels. Both NeRF-based approaches (i.e. Semantic NeRF and NeSF) succeed at this task by predicting, in addition to per-point radiance and density, the point's semantic class. However, both approaches are limited. Specifically, Semantic NeRF does not generalize to novel scenes. More- over, NeSF neglects color information, is not validated in standard dataset, and requires test-time optimization to generalize to novel scenes. On the other hand, Semantic SRN conducts unsuper- vised feature learning from posed RGB images and then learns to output semantic segmentation maps. Compared to these works, our method generalizes to novel views and objects, does not rely on per-scene test-time optimization. requires only relative camera poses, and is validated on well- established benchmark (i.e. PartNet).
2.3 3D RECONSTRUCTION FROM IMAGES
Multiple-view reconstruction. Reconstructing a 3D scene from a limited set of 2D images is a long-standing challenge in computer vision (Hartley & Zisserman, 2003). Modern learning-based methods (Hartmann et al., 2017: Donne & Geiger, 2019; Huang et al., 2018) can reconstruct with just few views by leveraging shape priors learnt from large datasets. However, to perform such reconstruction, these methods need to train on costly and explicit 3D supervision. In contrast, Seg- NeRF can provide dense high-quality 3D geometry while only requiring 2D supervision.
Single-view reconstruction. Most single-view 3D reconstruction methods condition neural 3D representations images (Niemeyer al., 2020a; Xu et al., 2019; Liu et al., 2019a). For instance, Niemeyer et al. (2020a) uses an input image to condition an implicit surface-and-textum network. Liu et al. (2019a) learn deform an initial mesh into target 3D shape. Our SegNeRF method
3

---

Under review as a conference paper at ICLR 2023
shares this ability of reconstructing from single view, however, it can also leverage variable number views to improve the reconstruction.
METHODOLOGY
We now present SegNeRF, a multi- method for jointly learning consistent 3D geometry, appearance, and semanties. During training, require multiple posed views of each object along with their corresponding semantic maps. While SegNeRF trains solely with image-level supervision, inference time it receive single image of an unseen object and produce (i) 2D novel-view part segmentation, (ii) 3D part segmentation, and (iii) 3D reconstruction. Furthermore. SegNeRF can generate more accurate predictions by leveraging additional input posed images of the object.
Before detailing our method, SegNeRF, we first give an overview of volume rendering which allows us learn in 3D space from images.
3.1 OVERVIEW OF NEURAL VOLUME RENDERING
Neural volume rendering relies on learning two functions: (x; 0) R + R which maps a point in space onto a density o, and c(x, d;0)= R RS + RR that maps point space viewed from direction d onto radiance c. The parameters that define the density and radiance functions typically optimized to represent a single scene by using multiple posed views of the scene. order to learn these functions, they are evaluated at multiple points along ray r(t) o + td, t [t,, defined by the camera origin E R&, pixel viewing direction d, and camera near and far clipping planes in and ly. pixel color for the ray can then be obtained through volume rendering via:
C(r:0) T(t) a(r(t)) c(r(t), d) dt, where T(t) exp (-j (((()) ds )s).
(-jott(e)
(1)
which can be computed with the original posed images. In practice, summation discrete samples along the ray is used instead of the continuous integral.
This volume rendering process allows to supervise the learning of implicit functions and 0, which span 3D space by using only rendered images through the reconstruction loss:
Tree (0)
1 R
Σ c())
-(r;))
where the batch of rays generated from random subset of pixels from training images.
3.2 PREDICTING VOLUME RENDERING
In the previous section, minimizing the reconstruction loss in equation 2 yields an implicit repre- sentation for a single scene. possible to obtain a model that generalizes to novel scenes by conditioning the learnt density and appearance local feature vectors f(I, d) derived from each the input source images I, particular, given a set images is possible to predict the radiance and density functions as:
c(L,x, d) = Go E o d) Go
( T
(3)
LEI
where G. and Go are MLPs that predict radiance and density fields for object with images at a given location and direction. The same reconstruction loss in equation 2 is minimized to obtain network parameters for both the feature extractor and the MLPs G. and Go.
Next, we elaborate on the inner workings of the feature extractor f. Each input image 4 is first encoded into feature map E(4). Then, each query point projected onto every feature image, obtaining image coordinates (x) and leading to vir-view feature vectors E(T;)..6x) Query points also projected onto 7, local coordinates via its associated projection matrix p()) A positional

---

Under review as a conference paper at ICLR 2023
encoding is then applied onto the local coordinates, resulting in x(P()) The per-view feature vectors (x)))x((x) are concatenated with the positional-encoder local coordinates ((p())) and the viewing directions d. The concatenated vectors are then processed by an MLP H to obtain per-view feature vectors f[;, x, d) as follows:
f(4, x, d) = ((caa(((4))) (P()))
(4)
In summary, feature vectors across all imput views are aggregated obtain a single per-point feature vector x(I,x, d), which is fed to the global MLP layer G to predict appearance and geometry as shown in equation 3.
3.3 SEMANTIC VOLUME RENDERING
We now deseribe our proposed extension this framework, which enables our network learn both 2D and 3D semantic segmentation from object images. In a similar fashion to how radiance field represents radiance at each point in space, we can imagine similar semantic field which represents the likelihood of an object part occupying each point in space. We propose to learn such semantic field function s(I, x, d) E IR', where e the number of object part classes, using an MLP g, similar to equation 3. In particular, function predicts a probability score for each location and direction conditioned on set images Similar to the color rendering, we aggregate the semantic predictions along ray for given set of images by using the learnt density function follows:
S(,,r; / T(O T()) ((((() s(, d)dd.
A softmax function is used convert the aggregated semantic scores $ to pixel semantic probabil- ities. At last, the predicted pixel semantic probabilities are optimized using a cross entropy loss in the following form:
Esem (0)
1 [R
Σ ((:))) FER
(6)
where S(r) is the ground truth segmentation label for the pixel corresponding to ray r. We note that train SegNeRF jointly for segmentation and image reconstruction through the combined loss
Cior Cree (f) + XC (8).
We emphasize that, upon convergence, single forward pass through SegNeRF obtains the semantic field function which can be leveraged for point cloud segmentation applications. Experimentally. show that the semantic field s, learnt using only image supervision, contains accurate 3D seg- mentation.
3.4 3D SEMANTIC SEGMENTATION
Next, we show how to leverage the learnt semantic field function 8 for 3D semantic segmentation. The task of 3D semantic segmentation requires classifying every point
object's surface with a single class с out of a known set of classes C. In practice, methods typically aim to classify discrete of points X sampled from the object's surface. This classification can be achieved by querying the semantic field function sII x, d) at every point in X. The class for point x for novel object captured by set of images is given by:
c(x) arg max sII, x, 0).
Even though the semantic field function has been trained through volume rendering with image supervision, we show experimentally that the learnt field is consistent with the underlying 3D rep- resentation and thus be used in point-wise fashion for predicting 3D semantic segmentation.
5

---

Under review as a conference paper at ICLR 2023
Computing 3D semantic segmentation with this procedure is vastly more efficient than performing volume rendering, since classifying each point requires a single forward pass instead of aggregat- ing multiple computations per ray done for volume rendering. While the learnt semantic field represents semantic segmentation at every location in space, we extract segmentation at the same set of points X as point cloud segmentation methods for comparison purposes. However, this set of points is generally not required for computing the segmentation of novel object. In practice, it possible extract semantic segmentation for an object without having access set of points by first extracting an object mesh, before querying the vertices of the extracted mesh for semantic segmentation. We demonstrate this capability by obtaining semantic reconstruction of real objects in Section 5 from single image without access to any 3D structure.
4 EXPERIMENTS
We experiment with our novel SegNeRF framework, showing its capability to generate novel 2D views with segmentation masks, reconstruct 3D geometry and segment 3D semantic parts of objects from one or more images.
Datasets. For all experiments, we utilize a subset of the PartNet (Mo et al., 2019) dataset that overlaps with the ShapeNet (Chang et al., 2015) dataset. PartNet contains an extensive collection of object meshes that have been divided into parts, along with their corresponding human-labeled part semantic segmentation at multiple resolutions. ShapeNet includes the corresponding colored 3D meshes, which we require to obtain the input renderings for our models. From each matched PartNet and ShapeNet object model, we first align and normalize the part and object meshes. We then render views from the colored mesh along with part segmentation masks rendered from the aligned part meshes using the PyTorch3D (Ravi et al., 2020) internal rasterizer. For the training set, we render 250 camera perspectives sampled uniformly at random on a unit sphere, with the object at the origin. To generate the validation and test sets, we render 250 camera perspectives sampled from an archimedean spiral covering the top half of each object instance, following previous work (Kohli et al., 2020). Due the need for having both semantic parts and colored meshes, the number of annotated object instances is lower than in the original PartNet dataset keeping 81.2% of the shapes on average. We report experiments for objects with the highest number of instances and re-run the baselines on this PartNet subset for fair comparison. We will release the RGB and semantic renderings publicly for reproducibility.
Training Details. Following PixeINeRF (Yu et al., 202 we train our proposed SegNeRF models using a batch size of (objects) and 1024 sampled rays per object. We train all of our models on Nvidia V100 GPUs for 600k steps, with single model trained for each object category. We use segmentation loss weight of À =
4.1 2D NOVEL- VIEW PART SEGMENTATION
Given a single or multiple posed views of object unseen during training, we aim to obtain the segmentation mask for the given object at novel views. This task requires not only understanding the semantics of an object class in 2D space but also in 3D space ensure consistent novel-view segmentation.
Metrics. We evaluate novel-view segmentation by using mean intersection over union (mIoU) over every part category for each object class. For single-vier view experiments, a front-facing image of the object (view ID 135) is used as input. total of 25 novel views around each object are predicted and averaged for each model. These 25 views are equally spaced around the the archimedean spiral.
Baselines. For the PartNet 2D semantic segmentation task, we compare quantitatively and qualita- tively with three baselines. First, we establish an upper bound by training a DeepLabv3 model in an unrealistic setting in which the model is granted access to all original observations, and performs segmentation the test images without novel view prediction. Second, Semantic-SRN (Kohli et al., 2020), a method proposed by Kohli er al. which is the current ofate-of-the-ar in few- shot novel-view semantic segmentation. but requires test-time optimization. Lastly, proposed "PixelLab", a two-stage baseline composed of PixeINeRF (Yu et al., 2021b) for novel view synthe- sis and Deepl Lbv33 (Chen al., 2017) for semantic segmentation. We use DeepLabvv, a popular

---

Under review as a conference paper at ICLR 2023
Avg
Bed
Bott
Chair
Clock
Dish
Disp
Ear
Fanc
Kndee
Microo
Stora
Table
Trush
Vase
DeepLabv3
23.88
24.36
38.12
PaxelLab
21.45
13
27.95
12.79
18.17
16.66
45.80
18.37
28.25
26.84
19.35
15.48
10.2
20.79
26.4
Semantic SRN
12.81
8.09
SegNerf
19.65
25,66
17.13
Table I: Fine-grained 2D semantic segmentation for novel views (2D part-category mloU %% ). Quantitative results for fine-grainets 3D semantic segmentation are presented, with detailed break- down by category. Our method outperforms the baseline methods by significant margins in most categories. Upper bound method (DeepLabv3) is highlighted in gray.
Source
Targel
GT
Ours
Semantie-SRN
Source
Target
GT
Ours
Semantie-SRN
Figure 2D semantic part segmentation for novel views. Qualitative results for 2D semantic part segmentation for novel views.
image semantic segmentation baseline, with a ResNer50 backbone pre-trained on ImageNet and finetuned on our PartNet subset for 20 epochs using ADAM and a base learning rate of 1e-3. Once DeepLab v3 trained, we evaluate it on the validation set generated by PixeINeRF consisting of 25 novel poses, and rendered from single view of the object instance using category-specific PixelNeRF. We refer the reader to the Appendix for further details.
Results. Quantitative. In Table 1, show the results of 2D semantic segmentation for novel views. Our method exhibits better performance than Semantic-SRe and "PixelLab" for all cate- gories. Lastly, expected, the DeepLabv3 upper bound performs slightly better than SegNeRF (mloU 31.59% vs. 30.30%). The results of the coarse coarse-grained 2D semantic segmentation for novel views are described in the Appendix. Qualitative. Semantic-SRN produces undesirable sults, as shown in Figure 2. We attribute this phenomenon to Semantic-SRN's optimization relying only on RGB information, which does not match the model's training on RGB and semantic maps, introducing noise in the objects' latent code. Furthermore, Figure 2 also illustrates how PixelLab mislabels fine structures. Unlike PiXeINeRF, SegNeRF does exploit the 3D representation while training. making it more apt for directly predicting semantic labels.
4.2 SINGLE/MULTA VIEW 3D PART SEGMENTATION
The task of 3D part segmentation aims to assign part category to each point in a given 3D point cloud representing a single object. Here, we aim to perform 3D part segmentation of given point cloud by using only the information from specific posed images. The point cloud is only used as query for segmentation to compare against point-basee methods.
Metrics. We evaluate 3D part segmentation by calculating the mloU over part categories for every object class. For single-view experiments, front-facing image of the object (view ID 135) used as input. Experiments using two images take as additional back-facing image (view ID 192), while experiments with four images include a top and a front view (view IDs 0, 249). Additional metrics such accuracy and mean accuracy are reported in the Appendix.
Baselines. We compare our method with a fair multi-view baseline and more widely adopted 3D point-based methods. For the point-based methods, train PointNet (Qi et al., 2017a), Point- Net++ (Qi et al., 2017b), and PosPool (Liu et al., 2020) on our dataset. Additionally, we propose multi-view (MV) baseline in which query point cloud is projected into DeepLabv3 predicted semantic segmentation masks. Every point is projected onto each view to obtain a per-view segmen- tation label. It is important to mention that the final label is computed by voting method among
7

---

Under review as a conference paper at ICLR 2023
Avg
Bed
Bolt
Chair
Clock
Dish
Disp
Ear
Fanc
Knile
Micro
Stora
Table
Trush
Vase
NeintNet
13.26
18.71
39.1
NointNet+4
34.88
29. 16
21.78
39:71
25.97
41.45
62.4
19.37
42.94
27.91
29.55
55.78
34.75
33.89
43.65
PosPool
46,42
43.2
25.5
48.46
41.44
62.86
66.53
27.18
50.5
31.19
54.99
47.04
43.99
52.11
54.95
MV
22
12.08
36.01
24.41
18.38
14.9%
36.41
[5.31
34.12
25.43
17:29
13.46
11:45
18.96
29.7
MV (2)
25.16
12.96
37.61
31.09
19.29
13.78
51.39
17.21
37.65
27.13
16.04
16.6
16.03
22.33
33.16
MV
25.25
15.38
30.82
34.2
16.71
11.47
53.13
18.04
37.65
30.63
14.76
15.96
17.09
22.94
34.44
MV (25)
27.43
18.21
37.91
36.26
19.12
15.1
53.37
24.89
40.74
27.46
15:35
21.3
17:29
22.16
34.79
SegNeRF (1)
32:44
15.99
41.05
34.83
24.09
39.73
57.43
21.36
44,25
41,08
24.79
22.71
15.58
30.44
40.87
SegNeRI
36.09
18.93
42.67
37.93
26.45
44.98
61.83
26.69
49.1
47.3
23.61
26.73
19.42
31,98
47.67
SegNoRF
37.46
21.1
40.04
29.44
62.49
24.12
49.69
27.38
23,08
32,79
47.43
Table 2:
Fine-grained 3D semantic segmentation (3D part-category mloU%). Quantitative re-
sults for fine- ine-grained 3D semantic segmentation are presented, with a detailed breakdown by cate- gory. Our method outperforms the 2D-supervised methods by significant margins in all categories, and attains comparable results with 3D-supereised methods. Methods that have access to 3D super- vision ppoint-based methods) are highlighted in gray.
Source
Ours
PosPool
MV
Source
Ours
PosPool
MV
Figure 3:
3D Semantic Part Segmentation from Multiple Images. Qualitative results for 3D semantic part segmentation from four input source views.
views, and occluded points in each view are not considered in the voting, For points occluded in all views, we take the label of the nearest neighbor point that has been seen at least in one view. MV is more comparable baseline than point-based methods, since it is also trained exclusively using image supervision. Note that MV (25 images) is an upper bound baseline, since it has access to all semantic segmentation observations while our model only needs couple of views.
Results. Quantitative. Table 2 reports mIoU for all methods. We find that SegNeRF performs better than all methods that are supervised only with images (bottom of table). In particular, SegNeRF out- performs all such baselines in every single class, suggesting that 3D awareness, as delivered through volume rendering, leads to improved performance. Interestingly, even when comparing MV's 25- image upper bound against SegNeRF's l-image lower bound, we find that SegNeRF obtains better average performance. Moreover, when compared point- based methods (bottom of the table)- and although SegNeRF lacks access to 3D supervision. we find that it performs comparatively well. Specifically, SegNeRF outperforms two point-based methods (i.e. PointNet and PointNet+++) on av- erage, and even outperforms the top-performing method (PosPool) on two classes. Qualitative. We show qualitative results in Figure 3. As expected, access to texture information delivers consistent improvements in segmentation performance on textured parts. This phenomenon is illustrated in the bottle and knife in Figure 3. PosPool struggles to segment the bottle's neck and the knife's sheath while SegNeRF segments these parts seamlessly. Note that PosPool and point-based methods in general have an advantage in this dataset due to the existence of points enveloped within objects, as seen with the green plate inside the microwave in Figure 3.
8

---

Under review as a conference paper at ICLR 2023
5 DISCUSSIONS
Volume vs. Surface Rendering. The task of semantic segmentation requires predicting semantic classification of points the surface of opaque objects. It intuitive to attempt to obtain semantic segmentation from a single point on the object surface rather than volumetric integration along the whole ray that considers the possible existance of semi-transparent volumes. Inspired by this intuition, we experimented with replacing the volume rendering formulation shown in equation with surface rendering. However, due to the sparse supervision space given by surface rendering, we found that volume rendering leads to better performances. We defer the details and experiments on surface rendering to the Appendix.
SegNeRF with single images in the wild. Finally, we experiment with images taken in the wild to assess our model's s generalization ability in performing both reconstruction and segmentation from single image. Images pre-processed to remove the background using off-the-shelf method since we train with synthetic renderings using white backgrounds. We show the 3D reconstruction and part segmentation results in Figure 4. We observe that model has learnt to reconstruct parts that aren't 't visible in the images such as the back legs. The delicate structures of the chair, such as the office chair's wheels and the garden chair's holes, recovered. However, even if the office chair is not perfectly reconstructed, our method's segmentation ability unaffected. Our method continues to provide accurate part segmentation.
Sample Views
Real Images
Input image
Pre-pracessed image
Output reconstruction
Figure 4: 3D reconstruction and part segmentation on single-view real images. Qualitative results of two chairs given single-view images. From left right: Original image, pre-processed image with background removed, semantic reconstruction from three different points of view: input view, front view, and back view. Notice the ability to recover and segment Ene-detailed structures as well as unseen parts such as the back leg of the second chair.
6 CONCLUSION
In this work, we presented SegNeRF. model for predicting joint appearance, geometry, and seman- tic fields for objects supervised only with images. We observe that NeRF-like representations are capable of learning more than radiance fields through volume rendering with image supervision, as shown with the semantic fields. We demonstrated the efficacy of SegNeRF on the tasks 2D novel view segmentation and 3D part segmentation from single and multiple images. From single image in the wild of a single object, we demonstrate a practical application of SegNeRF in performing a semantic reconstruction which could be valuable for autonomous navigation. We envision future work investigating segmentation at a larger scale (e.g. scene segmentation) and embedding more than just semantic information (e.g. instance-level segmentation).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International conference on machine learning, pp. 40-49. PMLR, 2018.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprini ar0iv:/608.044 04236, 2016.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information 3D Model Repository. Technical Report arXiv:1512.0301 [cs.GR]. Stanford University - Princeton University - Toyota Technological Institute at Chicago, 2015.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 2017.
Simon Donne and Andreas Geiger. Learning non-volumetric depth fusion using successive reprojec- tions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7634-7643, 2019.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object recon- struction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017.
Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape induction from 2d views multiple objects. In 2017 International Conference 3D Vision (3DV), pp. 402-411. IEEE, 2017.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Wilffiid Hartmann, Silvano Galliani, Michal Havlena, Lue Van Gool, and Konrad Schindler. Learned multi-patch similarity. In Proceedings of the IEEE international conference on com- puter vision, pp. 1586-1594, 2017.
Po-Han Huang. Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2821-2830, 2018
Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 12949-12958 2021.
Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 3d shape segmentation with projective convolutional networks. In proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3779-3788, 2017.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 371-386, 2018.
Amit Pal Singh Kohli, Vincent Sitzmann, and Gordon Wetzstein. Semantic implicit neural scene representations with semi-supervised training. In 2020 International Conference on 3D Vision (3DV), pp. 423-433. IEEE, 2020.
Truc Le and Ye Duan. Pointgrid: A deep network for 3d shape understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9204-9214, 2018.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. pointenn: Convolu- tion on x-transformed points. Advances in neural information processing systems, 31, 2018.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for based reasoning. In Proceedings of the IEEE/CVF International Conference on Com- pufer Vision (ICCV), October 2019a.
10

---

Under review as a conference paper at ICLR 2023
Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning infer implicit surfaces without 3d supervision. Advances in Neural Information Processing Systems, 32, 2019b.
Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look local aggregation operators in point cloud analysis. ECCV, 2020.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution surface construction algorithm. ACM siggraph computer graphics, 1987.
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collec- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7210-7219, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
cupancy networks:
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object under- standing. In Proceedings of the IEEF/CVE Conference on Computer Vision and Puttern Recogni- tion (CVPR), June 2019.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. arXiv preprint arXiv:2201.05988 2022.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020a.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3504-3515, 2020b.
Michael Oechsle, Songyou Peng. and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In International Conference Computer Vi- sion (ICCV), 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CV Conference Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings the and pattern recognition, pp. 652-660, 2017a
the IEEE conference computer vision
Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint 1XXvv:700.004113 2017b.
Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black. Generating 3d faces using convolutional mesh autoencoders. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 704-720, 2018.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin John- son, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXivv2007.08501, 2020.
11

---

Under review as a conference paper at ICLR 2023
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein, Scene representation networks: Con- tinuous 3d-structure-awwe aware neural scene representations. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Yafei Song, Xiaowu Chen, Jia Li, and Qinping Zhao. Embedding 3d geometric features for rigid object part segmentation. In Proceedings the IEEE International Conference Computer Vision, pp. 580-588, 2017.
Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes, arXiv preprint arXiv:2 13260, 2021.
Zongji Wang and Feng Lu. Voxsegnet: Volumetric cnns for semantic part segmentation of 3d shapes. IEEE transactions visualization and computer graphics, (6(9):2919-2935)) 2019.
Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Citynerf: Building nerf at city scale. arXiv preprint arXiv:2112.05500 2021.
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in Neural In- formation Processing Systems, 32, 2019.
Lior Yariv, Matan Atzmon, and Yaron Lipman. Universal differentiable renderer for implicit neural representations. Advances Neural Information Processing Systems, 2020.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering neural radiance fields. Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 5752-5761, 2021a.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021b.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene la- belling and understanding with implicit scene representation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 15838-15847, 2021.
A DATASET DETAILS
We present the abbreviation for some of the categories reported in the main manuscript in Table
Abbreviation
Bott
Dish
Disp
Ear
Micro
Full Name
bottle
dishwasher
display
earphones
faucet
microwave
storage furniture
Table 3: Category abbreviation. manuscript.
The complete name for the reported categories in the main
We give additional details the number of distinct shape instances remaining from each category remaining after matching objects from PartNet with their corresponding objects ShapeNet in Table 4. We maintain above 90% categories such chair, clock, display, faucet, storage furniture, table, trash, and vase. On the other hand, these categories are below 90%: bed, bottle, dishwasher, earphones, knife, and microwave.
Total
Bed
Bolt
Chair
Clock
Dish
Disp
Ear
Fauc
Knife
Micro
Stora
Table
Trush
Vase
PartNet
22588
248
519
6400
579
201
954
247
708
384
212
2503
8309
340
1104
Ours
21468
192
435
6311
552
82
927
648
326
120
2243
8179
321
1064
Percentage
95.3
774
838
98.
95.3
40.7
978
21.5
91.3
849
566
97.3
98.4
94.4
96.4
Table 4: PartNet subset statistics. Number of distinct shape instances for the complete PartNet dataset and the subset utilized in this paper. Additionally, the percentage of the comparison between the complete PartNet dataset and ours.
12

---

Under review as a conference paper at ICLR 2023
The categories that are not taken into account for the experiments are: bag, bowl, door, hat, key lamp, laptop. mug, fridge, and scissors. The total number of instances that are within these categories is 4163 (15% from the original PartNet dataset).
B TRAINING DETAILS
B. SEGNERF ARCHITECTURE
We follow the training procedure from Yu et al. (2021b), the encoder & is ResNet34 backbone and extract feature pyramid by taking the feature maps before the first pooling operation and after the first ResNet 3 layers. Network H. is composed of 3 fully connected layers while network is made up of fully connected layers. All layers use residual connections and have a hidden dimension of 512. We use ReLU activations before every fully connected layer. The positional encoding comprises six exponentially increasing frequencies for each input coordinate.
We train for 600000 iterations, which took roughly 6 days on 4 V100 NVIDIA GPU.
B.2 2D NOVEL-VIEN PART SEGMENTATION
DeepLab v3. We train one model per category for 20 epochs (max. one day in 4 VI00 NVIDIA GPUs) with a batch size of 100. Additionally, the model expects input normalized images, i.e. mini- batches of 3-channel RGB images of shape (N, 3, H, W). where is the number images, H and W are the height, and width of the images. The images have to be loaded in to range of [0, 1] and then normalized using mean [0.5, 0.5, 0.5] and std [0.5, 0.5, 0.5].
DeepLab v3 returns OrderedDict with two Tensors of the same height and width as the input Tensor, but with 21 classes. output["out contains the semantic masks, and output[ aux'] contains the auxiliary loss values per-pixel. We remove the last layer of 21 classes and replace it with the number per category. In inference mode, output[ aux") ] is not useful. So, output[ out'] is shaped (N, C, H. W). C being the classes per category and the desired resolution. The resolution was the finest level (3).
We utilize ResNet-50 backbone, and the model was pre-trained on COCO 2017. Lastly, we use Adam optimizer, and cosine annealing LR scheduler with T' maz = 10 were used.
B.3 SINGLE/MULTI-VISS 3D PART SEGMENTATION
PointNet. and PointNet++. We follow the training procedure from Qi et al. (2017a) and Qi et al. (2017b), respectively. We train only one model for all categories for 251 epochs (2 days in V100 NVIDIA GPU). The batch size was 45 and 25 for PointNet and PointNet++ models, respectively. Additionally, each model expects as input normalized point clouds, i.e. mini-batches of objects in a range of [0, 1]. We utilize Adam optimizer with learning rate 0.001, betas of (0.9, 0.999) and weight decay of 0.5. Both models were initialized with Xavier uniform initialization. Lastly, use PointNet++ MSG version.
PosPool. We follow the training procedure from Liu et al. (2020), and train their architecture variant which uses sinusoidal positional embedding, PosPool".
B.4 SINGLE/MULTI- 3D RECONSTRUCTION
SoftRas. We follow the training procedure from Liu et al. (2019a), we train one model for chair, table, and all categories. In the main manuscript, the reported results are with chair and table models that were trained for 250000 iterations. Here, we detail results for all categories model that was trained for 500000 iterations. We utilize Adam optimizer with learning rate 0.0001, betas of (0.9, 0.999).
13

---

Under review as a conference paper at ICLR 2023
Source
Ours
PixeNNRRP
DVR.
SoftRas
Source
Ours
PixelNeRF
DVR
Roll
Figure 5: 3D Reconstruction from Single Image. Qualitative results for 3D reconstruction presented for the chair and table categories.
C 3D RECONSTRUCTION
Obtaining a 3D reconstruction of an object from a single or few images is an ill-posed problem since there be infinite ways in which occluded portions of object could be structured in 3D. However, most objects tend to follow similar distributions amongst their class. For example, chairs tend to have four legs, a seat, and backrest. Thus, neural models should be able generate 3D model from a single observation of an object by learning object class distributions.
In real-wwa world scenario, Thus, if one would like to use our method to get the 3D part segmentation of an object in the wild, it necessary first to obtain the 3D reconstruction from the taken views, which can also be obtained with SegNeRF. After having the point cloud, each point can be queried in SegNeRF to obtain its segmentation prediction. Thus, this section shows an extensive 3D reconstruction task analysis in which SegNeRF could be helpful in real-world scenarios.
more likely to have access to a set of images rather than point cloud.
For the task of 3D reconstruction, we generate an object's 3D mesh given a set posed input images of the object. To generate this mesh follow the standard practice in constructing set of query points as a grid within a volume at a defined input resolution (Mildenhall et al., 2020) detailed in Figure 1. In this work, improve the extracted representation by leveraging the predicted semantic field, thus removing the predicted points as background by setting their density to 0. A mesh then be extracted from this grid by applying density threshold, and then executing marching cubes (Lorensen & Cline, 1987).
Single- View Reconstruction. Here we show that model is capable of learning such distributions for a known object class, and can generate good 3D reconstructions. We compare against Soft- Ras (Liu et al., 2019a) and DVR (Niemeyer al., 2020a). two popular 2D-supervised single-view reconstruction methods. We follow each of their training procedures using our training data. A single model trained for each category. Quantitative results such as chamfer distance, f-score, precision, and recall are reported in the Appendix.
Figure provides evidence that PixelNeRF and our method can reconstruct high-quality geometry for complex objects, unlike existing single-view reconstruction models. Both methods can learn intrinsic geometric and appearance priors, which are reasonably effective even for parts not seen in the observation (e.g. chair and table legs). Similarly, SoftRas performs poorly in producing thin parts of the objects since it deforms a sphere mesh a fixed size.
It also could be observed that the PixeINeRF reconstruction can provide finer details than our method. For example, the table reconstructed by PixelNeRF on the bottom-lef side has a leg size more consistent with the object. Equally, some chair parts missing in SegNeRF reconstructions. While our method does not achieve the same performance as the PixeINeRF baseline, we believe that the 3D segmentation capability comes at the price of slightly worse reconstruction.
14

---

Under review as a conference paper at ICLR 2023
Chair
Table
CD 4
F-Sc f
P
R
CD 4
F-Se
P
R
SegNeRF (1)
3.82
79.31
75.18
84.72
5.07
79.84
75.29
86.16
UniSeg (1)
2.70
72.4
75.99
70.23
3.45
71.26
87.43
62.41
SegNeRF (2)
1.73
86.53
84.94
88.72
1.74
85.96
83.79
88.92
UniSeg (2)
2.21
78.91
83.21
75.9
2.79
77.76
89.18
70.75
SegNeRI (4)
1,54
90.06
90.21
90.28
1.48
90.33
89.99
91.08
UniSeg (4)
2.38
79.13
85.68
74.29
3.09
77.56
90.87
69.63
Table 5: 3D Reconstruction Surface Rendering vs. Volume Rendering, Quantitative results for 3D reconstruction are presented, with the varying number of imput images represented by the number in parentheses. Volume rendering results in better reconstruction for multi-image settings, while surface rendering results in higher precision single-image reconstruction at the cost of lower recall. Metries set key: CD: Chamfer distance., F-Sc: F-Score, P: Precision, R: Recall.
Source
Ours
PixeINeRtF
UniSeg
Source
Ours
PixelNeRF
UniSeg
Figure 6: 3D Reconstruction from Multiple Images. Qualitative results for 3D reconstruction from four images are provided for the chair and table categories.
D ADDITIONAL RESULTS
Volume vs. Surface Rendering. We present variant of our method dubbed UniSeg, which relies on surface rendering instead volume rendering to generate colored and semantic images. The task of semantic segmentation requires predicting classification only on object surfaces. It is intuitive to attempt to obtain semantic segmentation from a single point on the object surface rather than an integration along the whole ray that emanates from each pixel. Inspired by (Oechsle et al., 2021), we experiment with replacing the volume rendering formulation shown equation 5 with surface rendering. The density becomes discretized into binary occupancy representation for surface rendering. Rendering then becomes an assignment of the first surface value encountered as shown in equation 9, instead of having contributions to the final field values from all positions along a ray.
S(r)= - Σ s (r(ti)) o (r(t.) II (r(t.)) jai
(9)
Results. Although surface rendering should intuitively benefit semantic segmentation in 3D, we observe the opposite. When using surface rendering, the average mloU across all object categories is lowered from 32.44% to 26.92% (-5.51%) for the single image setting. As shown in Table 5 and Figure 6, the reconstruction recall (R) tends to be lower when using surface rendering while the pre- cision (P) is comparable. The lower recall and higher precision can be observed in the qualitatively thinner and smoother surfaces generated by the surface rendering method. Inspecting the Chamfer Distance (CD), we observe that the volume rendering benefits from more views, while the surface rendering saturates after two views. We believe that the full volume rendering has more learning capabilities than the surface one.
15

---

Under review as a conference paper at ICLR 2023
D.1 2D NOVEL- VIEW PART SEGMENTATION
In Figure 7 and Figure 8, we show additional result for table and chair categories. These examples randomly selected.
SOAP
Semantic SRN
Semantic
Source
Target
GT
Ours
Source
Target
GT
Ours
SRN
Figure 7: Additional 2D semantic part segmentation for novel views (table) results. Qualitative results for 2D semantic part segmentation for novel views.
Semantic SRN
Semantic SRN
Source
Target
GT
Ours
Source
Target
GT
Ours
Figure 8: Additional 2D semantic part segmentation for novel views (chair) results. Qualitative results for 2D semantic part segmentation for novel views.
16

---

Under review as a conference paper at ICLR 2023
D.2 SINGLE/MULTIVIEE VIEW 3D PART SEGMENTATION
In Figure 9, we show additional result for all categories. These examples were randomly selected.
Source
GT
Ours
PosPool
MV
Source
GTr'
Ours
PosPool
MV
Figure 9: Additional 3D Semantic Part Segmentation from Multiple Images Results. Qualitative results for 3D semantic part segmentation from four input source views.
Table 6 and Table 7 report accuracy and mean accuracy for all methods.
17

---

Under review as a conference paper at ICLR 2023
Avg
Bed
Bott
Chair
Clock
13ish
Disp
Far
Fauic
Knife
Mileoo
Storn
Tablle
Trash
Vase
MV (2)
65.48
29.87
92.39
68.69
56. 16.26
68.61
92.71
53.75
09
63.84
75.95
47.12
63.86
52.60
83.01
MV
66.06
33.18
89.72
71.75
55.55
67.15
92.97
55.42
67.50
66.98
78.87
46.23
63.49
52.71
83.35
MV (25)
69.00
34.44
92.94
74.54
60.25
74.02
93.81
62.99
71.09
62.87
82.13
54.20
53.94
84.26
(1)
70.70
33.15
94.01
70.97
64.10
83.38
92.50
64.54
72.23
74.21
78.74
58,91
63.64
54.95
$4.38
SegNerf (2)
72.76
37.49
94,21
73.03
64,35
84.57
94.05
66.72
74.63
77.60
79.15
63.49
67.05
57.10
85,23
SegNerf (4)
73.42
39.58
94.5
74.37
64.45
85.6
94,30
65.18
74.92
79.49
77.32
64.47
68.89
59.32
85,55
PointNet
76.09
45.55
89.36
74.70
71.54
88.23
96.10
65.67
72.97
58.05
89.33
74.75
73.12
74.66
91.44
27
67.82
77.04
PosPool
79.88
63.14
90.94
80.08
72.18
86.74
96.28
70.64
77.24
67.38
90.85
80.44
75.03
78.97
88.49
Table 6: Fine-grained 3D semantic segmentation (3D part-category accuracy %). Quantitative results for fine-gre inn-grained 3D semantic segmentation are presented, with a detailed breakdown by cat- egory. Our method outperforms the 2D- 2D-supervised methods by significant margins in all categories, and attains comparable results with 3D-supervised methods.
Avg
Bed
Bott
Chuir
Clock
Dish
Disp
Ear
Fauc
Knife
Micro
Stora
Table
Trush
Vase
31.85
25.
MV (2)
37.90
23.53
53.18
40.81
25.87
18.39
58.
37.31
50.97
39.95
21.64
24.57
38.60
34.56
62.68
MV
35.98
25.01
38.93
44.31
23.14
15.97
59.72
30.72
51.08
44.84
18.33
24.08
35.62
34.33
57.68
(25)
(1) (1)
45.52
32.06
46.19
49.44
32.16
58.97
67.78
18.95
58.42
55.23
40.12
33.53
24.22
47.00
53.17
SegNerf
49.35
35.76
47.42
51
34.34
61.85
73.08
63.38
37.64
37.08
29.69
47.64
$4.53
(eg
50.55
39.31
49.58
53.65
38.98
64.45
73.73
57.61
62.99
61.98
32,99
37.4
33.85
47.76
53.41
PointNet
45.16
35.31
33.39
45. 64
34,
$2.07
83.54
40.71
$8.49
33.50
34
43.03
40.98
46.29
52.54
PointNet++
PosPool
58.83
59.76
30,55
59.98
51.17
76.78
73.96
55.51
63.2
42.22
67.66
58.64
54.24
65.86
64.06
Table 7: Fine-grained 3D semantic segmentation (3D part-category mean accuracy %). Quan- titative results for fine-grained 3D semantic segmentation are presented, with detailed breakdown by category. Our method outperforms the 2D- -upeuerres methods by significant margins in all cat- egories, and attains comparable results with 3D-supervised methods.
D.3 SINGLE/MULTI- VIEW 3D RECONSTRUCTION
In Figure 10 and Figure 11, we show additional result for table and chair categories, These examples were randomly selected.
Source
Ours
PixeINeRF
DVR.
Rast
Source
Ours
PixeINeR
DVR
SoftRas
Figure 10: Additional 3D Reconstruction from Single Image Results. Qualitative results for 3D reconstruction are presented for table.
18

---

Under review as a conference paper at ICLR 2023
Source
PixeINeRP
DVR
Soft
Ours
PixeINeRF
DVR
SoftRas
Figure 11: Additional 3D Reconstruction from Single Image Results. Qualitative results for 3D reconstruction presented for chair.
D.4 SEMANTIC RECONSTRUCTION
We combine 3D reconstruction and semantic segmentation by first reconstructing objects into mesh. Vertices from the mesh are then colored according to the 3D segmentation predicted by SegNeRF at each of the vertex locations. The resulting a semantic reconstruction mesh which can be segmented into each of the object's parts. Please see the supplementary video for visualizations on semantic reconstructions of chair objects from source images, which showcases the ability of SegNeRF to perform joint 3D reconstruction and segmentation.
19


Reviews:
[{"review_id": "wTH0Zrb3nF", "reviewer": "Reviewer_zVwm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an extension of PixelNeRF to extend the neural field with auxiliary dimensions, corresponding to semantic classes of the scene. The authors propose to learn semantic fields using 2D semantic supervision and demonstrate that their approach reaches the performance of several prior approaches enjoying 3D semantic supervision (except one SOTA method).", "summary_of_the_review": "The paper provides an important study of leveraging 2D semantic maps for learning 3D-aware semantic representations for semantic novel view synthesis using NeRF. The method seems like a small extension on top of PixelNeRF, but there are a few points to be clarified (see Weaknesses). The empirical study looks good, but may not present too much insight.", "strength_and_weaknesses": "The paper is well written, the delivery is good, however, there are a few questions for the authors.\n\n- Fig.1 is nice but does not serve the purpose of explaining the method. Color coding segmentation/reconstruction/violet objectives are counter-intuitive. The 6 H/G MLPs are not placed into the context. The chair is presented from 3 different perspectives, yet only two are present on the left side. Without the introduction of the notation, the E symbol reads as taking expectation, and not the feature map explained deep in the Method section. I would recommend a complete re-design of the figure. \n\n- Sec. 2.1. states that voxelized representations have a cubic complexity in dimensions; this is not true in the light of works like TensoRF and other decomposition methods.\n\n- Sec. 3.2. is titled \"Predicting Volume Rendering\" - what does this mean exactly? Perhaps some rephrasing is in order.\n\n- Starting the \"Next, we elaborate\" paragraph in Sec. 3.2 and until the end of that section, the writing is rather negligent with notation. I would recommend checking how a similar concept was introduced in PixelNeRF and using a similar notation, more space to explain this rather critical part of the method and introduce notation before or around the first usage. In the end, I was not quite sure if the method (its part explained in the section) has any important differences from PixelNeRF, apart from the added semantic classes prediction. If not, then using PixelNeRF notation is advised.\n\n- Related to the previous comment, PixelNeRF applies positional encoding to 3D point x, whereas in your paper $\\gamma$ seems to be applied to the projected coordinates. Why was this change introduced, what was the motivation? Have you tried keeping this as in PixelNeRF?\n\n- In Sec. 3.3 you present the function $s(I, r, d)$, meaning that semantic information at each point depends on the viewing direction $d$. How so? Shouldn't it be independent of viewing direction like e.g. optical density $\\sigma$? \n\n- Why is there a zero in Eq. (8)? Is it by any chance there to fix the issue from the previous comment?\n\n- I'm not sure why Sec.3.4 is needed, if the function $s$ functions properly (not requiring Eq. 8 hack).\n\n- Why is there only 81% of PartNet dataset valid for dataset creation purposes?\n\n- View ID 135 must be clarified or moved to the supplmentary\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tfAJ534AsQ", "reviewer": "Reviewer_ynF6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a generalizable neural field representation that combines the traditional neural radiance field with a semantic field, which enables a simultaneous modeling the geometry, appearance and semantic information from a few images. The proposed representation enables 2D novel view segmentation and 3D part segmentation from single and multiple images. They also demonstrate the capability of 3D semantic reconstruction from single in-the-wild image.", "summary_of_the_review": "Given the above-mentioned weaknesses (limited contribution, missing analysis, etc), my suggestion is that the paper is currently marginally below the acceptance threshold for a publication.", "strength_and_weaknesses": "** Strengths **\n\n1. The proposed method/representation is quite concise and technically sound. Simply integrating a new semantic field into a NeRF framework can improve the modeling capability. It shows more possibility to extend from geometry and appearence to more information.\n2. They conduct extensive comparison studies to demonstrate the superior of the proposed approach in terms of semantic segmentation.\n3. Code and data will be shared and they would be a nice bonus to the literature.\n4. The paper is generally well written and easy to follow.\n\n** Weaknesses **\n\n1. The main weakness of the submission is the limited contribution it offers. The idea that integrating a semantic field into a traditional NeRF representation is straightforward and not novel enough. It looks more like an incremental work to NeRF and nothing special is proposed to address particular problems.\n2. Although the main focus of the paper is semantic segmentation, we would still like to know how it performs on geometry and appearance quantitatively. If possible, I would also like to see an ablation study to show whether adding a semantic field has any positive or negative effect on the original NeRF's geometry and appearance modeling performance.\n3. In Tab2, why using more views make the final reuslts worse for “Ear”, “Micro” and “Vase”? Could you provide a more detailed analysis of this observation?\n4. In Fig4, could you show more novel view synthesis results instead of only semantic segmentation? Why the top region of the chair back is recognized as purple?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m50n05MnhDT", "reviewer": "Reviewer_iUC9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to employ neural radiance fields to model the 3D part information of objects. In parallel with the color and density information, the NeRF model also predicts the semantic segmentation probability of each point in the 3D space. This yields a semantic field that enables the model to achieve good 2D and 3D segmentation results.", "summary_of_the_review": "The strength and weakness of this paper are clear. I don’t think this can reach the bar of ICLR because of the novelty of the idea and method.", "strength_and_weaknesses": "Strength:\n1. This paper is well written and the ideas are clearly illustrated.\n2. It is nice to investigate NeRF-based model for a discriminative task.\n3. It is good to see that SegNeRF achieves encouraging results on 2D and 3D segmentation tasks. \n\nWeaknesses:   \nThe major concern is the novelty of this work.\n1. For the task of 2D and 3D segmentation, introducing a semantic field is no longer a novel design. For example, SemanticNeRF[1] and PanopticNeRF[2] employed the segmentation field for Scene Labelling and Understanding, FENeRF[3] and sofgan[4] designed the semantic field for 3D face generation and editing. The motivation and method of this paper are highly similar to these previous methods.\n\n2. For the encoder-based frameworks, this paper combines ideas of IBRNet[5] and PointNeRF. \n\n3. I think the novelty is to apply previous ideas to the task of 3D part segmentation, but I don’t think this can reach the bar of ICLR.\n\n[1] In-Place Scene Labelling and Understanding with Implicit Scene Representation.  \n[2] Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation.  \n[3] FENeRF: Face Editing in Neural Radiance Fields.  \n[4] Sofgan: A portrait image generator with dynamic styling   \n[5] IBRNet: Learning Multi-View Image-Based Rendering  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RZVCXO5XRFp", "reviewer": "Reviewer_miT3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for 3D shape part segmentation. It learns an image-conditioned NeRF together with a semantic field. The method enables novel view (2D image) part segmentation and 3D point cloud part segmentation. The authors evaluate the proposed method on the PartNet dataset and show some real-world demos.", "summary_of_the_review": "I fail to tell much difference between the proposed method and existing methods (e.g., NeSF). I strongly suggest the authors provide a more thorough discussion and comparison with the existing semantic NeRF works. Otherwise, I would prefer to reject the submission.", "strength_and_weaknesses": "Strengths:\n1. The idea of training an image-conditioned NeRF together with a semantic field is simple and straightforward. \n2. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. The idea of training a semantic field in NeRF is not new. Many prior works (e.g., Semantic NeRF, NeSF, and Semantic SRN) share a similar idea, which limits the novelty of the proposed method.\n2. I think NeSF is very similar to the proposed method and fail to find any major difference between the two methods. However, NeSF is missing in the experimental comparison.\n3. In equation (5), why does the semantic field function s relies on the view direction d? From my understanding, the semantic label of a 3D point should not change with regard to the view direction. \n4. The current method is not tested within each object category. It would also be interesting to see whether the model trained on a shape category can generalize to similar categories as well, since we cannot train separate models for all shape categories in our real-world applications.\n5. Some experimental details are unclear or missing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject