Neural Field Classifiers via Target Encoding and Classification Loss
Authors: Xindi Yang
Zeke Xie
Xiong Zhou
Boyu Liu
Buhua Liu
Yi Liu
Haoran Wang
YUNFENG CAI
Mingming Sun
Venue: ICLR 2024 poster (2024)
Keywords: Neural Fields
NeRF
3D Vision
Scene Reconstruction
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Neural Field Classifiers via Target Encoding and Classification Loss can significantly outperform the standard regression-based neural field counterparts.

Abstract: Neural field methods have seen great progress in various long-standing tasks in computer vision and computer graphics, including novel view synthesis and geometry reconstruction. As existing neural field methods try to predict some coordinate-based continuous target values, such as RGB for Neural Radiance Field (NeRF), all of these methods are regression models and are optimized by some regression loss. However, are regression models really better than classification models for neural field methods? In this work, we try to visit this very fundamental but overlooked question for neural fields from a machine learning perspective. We successfully propose a novel Neural Field Classifier (NFC) framework which formulates existing neural field methods as classification tasks rather than regression tasks. The proposed NFC can easily transform arbitrary Neural Field Regressor (NFR) into its classification variant via employing a novel Target Encoding module and optimizing a classification loss. By encoding a continuous regression target into a high-dimensional discrete encoding, we naturally formulate a multi-label classification task. Extensive experiments demonstrate the impressive effectiveness of NFC at the nearly free extra computational costs. Moreover, NFC also shows robustness to sparse inputs, corrupted images, and dynamic scenes.

Published as a conference paper at ICLR 2024
NEURAL FIELD CLASSIFIERS VIA TARGET ENCODING AND CLASSIFICATION LOss
Xindi Yang" Zeke x
Xiong Zhou? Boyu Liu"
Buhua Liu?
Yi Liu" Haoran Wang?
Yunfeng Cai"
Mingming Sun?
"Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University
"Cognitive Computing Lab, Baidu Research
ABSTRACT
Neural field methods have seen great progress in various long-standing tasks in computer vision and computer graphics, including novel view synthesis and geometry reconstruction. As existing neural field methods try to predict some coordinate-basee continuous target values, such as RGB for Neural Radiance Field (NeRF), all of these methods are regression models and are optimized by some regression loss. However, are regression models really better than classification models for neural field methods? In this work, we try to visit this very fundamental but overlooked question for neural fields from a machine learning perspective. We successfully propose novel Neural Field Classifier (NFC) framework which for- mulates existing neural field methods as classification tasks rather than regression tasks. The proposed NFC can easily transform arbitrary Neural Field Regressor (NFR) into its classification variant via employing novel Target Encoding module and optimizing classification loss. By encoding a continuous regression target into high-dimensional discrete encoding, we naturally formulate a multi-label classification task. Extensive experiments demonstrate the impressive effectiveness of NFC at the nearly free extra computational costs. Moreover, NFC also shows robustness to sparse inputs, corrupted images, and dynamic scenes]
INTRODUCTION
Background Neural field methods emerge as promising methods for parameterizing field, rep- resented by a scalar, vector, or tensor, that has target value for each point in space and time. Neural field methods first gained great attention in computer vision and computer graphics, because baased neural field methods show impressive performance in novel view synthesis and sur- face reconstruction. Synthesizing novel-view images of 3D scene from a group of images is a long-standing task (Chen and Williams 1993 Debevec et al. 1996 Levoy and Hanrahan 1996 Gortler et al., 1996] Shum and Kang 2000) and has recently made sigmificant progress with Neural Radiance Field (NeRF) (Liu et al. 2020 Mildenhall et al. (021). Neural surface representation (Michalkiewicz et al. 2019 Niemeyer et al. 2020) Yariy et al. 2021 Wang et al.) 2021) is another important and loong-standing problem orthogonal to novel view synthesis.
NeRF Basics Without losing generality, we take a standard NeRF as the example. NeRF can efficiently represent a given scene by implicitly encoding volumetric density and color through coordinate- based neural network (often referred to as a simple multilayer perceptron or MLP). NeRF regresses
from a single 5D representation (a, y, z, 9, o)- 3D coordinates æ (a. y, 2) plus 2D viewing directions d= = (0, Φ)- to single volume density or and a view-dependent color e = (r, b). NeRF
approximates this continuous 5D scene representation with an MLP network fe (æ:; d) -> (c;
and optimizes its weights θ to map each input 5D coordinate to the corresponding volume density and directional emitted color.
For target view with pose, camera ray can be parameterized as r(() = o + id with the ray origin o and ray unit direction d. The expected color C(r) of camera ray r()) with near and far bounds i,
Equal Contrihutions; Correspondence to xiezeke @ bom.u.com. Code: http:///tti...cmmaaaaae///aaaaat

---

Published as a conference paper at ICLR 2024
and ły is
Ö(r)- - J (()()))()))
C(r)
where (t)) exp(- Jr ass)ds) denotes the accumulated transmittance along the ray from t, to t. For simplicity, we have ignored the coarse and fine renderings via different sampling methods.
The rendered image pixel value for camera ray can then be compared against the corresponding ground truth pixel color value C(r), through the N sampled points along the ray. Note that C (R, G, B) is the color vector and R. G, B E [0, 1] are the normalized continuous values, while the raw color values are integers in [0, 255]. The conventional rendering loss is the regression loss
L(
1
Σ
||Ō(r) -(r)).
(2)
R
PR
where || 1/2 is the lg norm, the ray/pixel r (ær, cr), and R is the training data (minibatch).
Motivation As the targets are continuous values previous studies, people naturally formulate neural fields as regression models. In this work, visit very fundamental but overlooked question: are regression formulation really better than classification formulation for neural field methods? The answer is negative. There exist overlooked pitfalls in existing neural fields. For example, NeRF and its variants output N points' predictions per pixel. Compared to classical supervised learning methods where each data point has its own ground-truth label, supervision signals for NeRF obviously very weak and insufficient. In the presence of weak or noisy supervision, neural networks may exhibit significant overfitting and poor generalization (Zhang al. 2017
Xie al. 2021a)
Contributions We mainly make three contributions.
1. We successfully design novel Neural Field Classifier (NFC) framework which formulates neural fields as classification tasks rather than regression tasks.
2. We propose Target Encoding and introduce a classification loss to transform existing Neural Field Regressors (NFR) into NFC. The implementation is quite simple, as we only need to revise the final layer of the original NFR and optimize a classification loss.
3. We are the first to explore regression versus classification for neural fields. Surprisingly, classification models significantly and generally outperform its conventional regression counterparts in extensive experiments the nearly free extra computational cost.
METHODOLOGY
In this section, we formally introduce the NFC framework and two key ingredients: Target Encoding and classification loss. Figure I Iilluttrates the structure of NFC and NFR.
Target Encoding and Decoding We first discuss our Target Encoding module and how revise existing neural field methods according to the encoding-decodiin rule.
There are various ways to encode an continuous value into discrete vector. For example, NeRF projects the input continuous coordinates into nigh-dimensiioal continuous vector via positional encoding before feeding the inputs into neural networks. Thus, this is kind of input encoding rule. Our target encoding requires projecting a color value into a discrete vector so that we can classify.
Suppose C (R, G, B) is the three-channel color. In the raw dataset without preprocessing, the color values are actually integers in [0, 255). For simplicity, we ignore the channel and consider single color value y E [0. 255). very naive target encoding rule that we directly use y as the class label. Then we quickly formulate 256-class classification problem via one-hot encoding for each channel. However, this one-hot target encoding rule is naive and inefficient for two reasons. First, the number of logits increases to 768 from 3, which can cost more memory and computational costs than the original simple MLP. Second, this naive target encoding ignore the relevant information carried by the classes. Suppose the ground-truth label of sample is 0. If model A predicts 1 and model B predicts 255, their loss will be equally high. This is obviously unreasonable, because is a much better prediction than 255.
2

---

Published as a conference paper at ICLR 2024
Regression
Rendering Loss
Input
Oulput
Volume Rendering
(ae, d)
MLP
C2
Volume Rendering
Target Decoding
C
+
C
ABCE
Classification
The illustration of standard Neural Field Regressors and our Neural Field Classifiers. Our method makes two modifications existing neural fields. First, due the Target Encoding module, the final output of neural networks need to be high-dimensional color encoding rather than three-channel color value itself. The designed encoding rule connects our high- high-dimensional discrete representation and the standard three-channel continuous representation. Second, we mainly use classification loss as the main optimization objective. Note that the classification loss, as the main optimization objective, can be larger than the standard MSE loss by two orders of magnitude.
Figure 1:
0
0
Target Encoding
0
0
0
203 110 179
0
Figure 2: The illustration of Binary-Number Target Encoding.
Thus, better target encoding rule desired for neural field methods. Fortunately, we discover that the classical binary-number system can work well as the target encoding rule for NFC. A binary number is a number expressed in the base-2 numeral system, a method of mathematical expression which uses only two symbols: O and 1. With such bimbry- encoding rule, express a color value y as an 8-bit binary number y, namely y = BinaryEncodd For example, EncoarryyEcod20000
[1, 1, 0, 0, 1, 1, 1] can serve the label vector for 8-label binary classification task. We illustrate the binary- number encoding in Figure
Finally, may let neural network to predict 0 or for each bit in the label vector long increase the number color logits from 3 to 24 and design a proper classification loss.
probabilistic losses, such Cross Entropy (CE). In this work, we also use an CE-based classification loss as the main optimization objective. In the numbry system, the place values increase by the factor of 2. We also need to let the class weight increase by the factor of as the place values.
Classification Loss Classification losses usually
We first formulate bit-wise classification loss
8
4. (ū. v) 255
2
(CE()O),00))
(3)
where j is the place index and ÿ is the predicted probability given by the final layer the MLP. We note that 2/-1 assigns higher weight the class with higher place value.

---

Published as a conference paper at ICLR 2024
An alternative choice is that first decode the predict probability ÿ back into continuous value Ö = 255 BinaryDecod g). We treat Ć as the weighted- predicted probability for channel and interpret the ground-trunt truth color value (E [0, 1) as the ground-truth probabilistic soft label. Then obtain channel wise classification loss
1 BinaryDecod(gj..
le(Č,) BCE(C, BCE
(4)
255
Which classification loss should we choose? According to our empirical analysis, we find that two classification losses both significantly improve exist neural fields, while the channel-wise classification loss given by (4) has a simpler implementation than the bit-wise classification loss given by (3). In the following of this paper, use the channel-wise classification loss the default classification loss unless we specify it otherwise.
We point out that, due the process of ray sampling and volume rendering, the predicted probability Ć (or ğ) of NFC does not strictly lie in (0, 1) like standard image classification model. In the case that the predicted probability is greater than or equal to one, the gradient of the classification loss may explode. This case is impossible in image classification but may happen in neural fields due to volume rendering. Thus, we slightly modify the classification loss as
Le (c, C* BCE(min(CC, c), C)2
(5)
where 0.001 is desired for the numerical stability purpose.
As the min(::) operation will vanish and does not update neural networks. To solve the gradient vanishing problem, let the standard Mean Squared Error (MSE) loss serve as a minor optimization objective. So the final optimization objective of NFC can be formulated as follow
-i-ddfferrntiable for the data points with C. & (0, 1), the gradient
LNFO(Ć, C) = í C ABCE(min(C11-c),,
(6)
where C
255 BinaryDecodd ( is the predicted color probability/value.
We note that the classification loss term is the main optimization objective, because the BCE- classification loss can be significantly larger than the standard MSE loss by more than one order of magnitude during almost the whole training process (after the initial tens of iterations). In practice, we do not need to fine-tune the hyperparameter e. Fine-tuning is easy, because the advantage of NFC over NFR is robust to a wide range of X (e.g. [0.1, 100)).
3 RELATED WORK
In this section, we review representative related works and discuss their relations to our method.
Neural Fields Physicists proposed the concept of fields to continuously parameterize an underlying physical quantity of an object or scene over space and time. Fields have been used to describe physical phenomena (Sabella 1988) and coordinate-basse computing image gradients (Schlüns and Klette 1997) and simulating collisions (Osher et al. 2004). Recent advances in computer vision and computer graphics showed increased interest in employing coordinate- based neural networks, also called implicit neural networks, that maps a 3D spatial coordinate to a flow field in fluid dynamics, Such neural networks are often referred to as neural fields (Xie et al., (202aa) or implicit neural representations (Sitzmann al., 2020, Michalkiewics
phenomena beyond physies, such as
colour and density field in 3D scene representation.
al. 2019) Niemeyer al. 2020; Yariv et al. 2021 Wang et al. (2021). Our previous work (Xie et al. 2023b)) proposed a novel loss but focused on employing structural information, while this work followed the experimental setting. To the best of our knowledge, previous studies did not touch classification framework of neural fields.
Target Encoding Target Encoding, also called Target Embedding, Label Encoding, or Label Embed- ding, has been studied in previous classification studies of machine learning (Bengio et al., 2010 Akata et al., 2013 2015; Rodríguez et al. 2018 Jia and Zhang 2021). However, previous studies only study how to project one discrete-label space into another discrete-label space for standard classification tasks. They failed to explore how to encode continuous targets for regression tasks into discrete targets for classification tasks. The comparisons between regression formulation and classification formulation of one machine learning task is also largely under-explored in related machine learning studies.
4

---

Published as a conference paper at ICLR 2024
4 EMPIRICAL ANALYSIS AND DISCUSSION
In this section, we conduct extensive experiments to demonstrate the effectiveness of NFCs over their standard regression counterparts.
We let the experimental settings follow original papers to produce the baselines, unless we specify it otherwise. The basic principle of our experimental settings is to fairly compare NFC and NFR. Thus, we keep all hyperparameters same for them. Our evaluation is also in line with the established image and geometry assessment protocols within neural fields community. More experiments details can be found in Appendix A
mainly choose four representative neural field methods as the backbones, including DVGO (Sun) al. 2022), vanilla NeRF (Mildenhall al.) 2021), D-NeRF (Pumarola et al.] 2021), and NeuS (Wang et al. (222). We present more quantitative results and supplementary experimental results of Strivee (Gao et al., (222) and 4DGS (Wu al.) 2023) in Appendix [ We implement the NFC variants by modifying the final layers' logits of the neural network architectures and the optimization objective. For simplicity, refer to NFC and NFR of backhone (e.g. NeRF) NeRF-R and NeRF-C, respectively.
4.1 NOVEL VIEW SYNTHESIS EXPERIMENTS
Ground Truth
Regression (Standard)
Classification (Ours)
DVGO NoRF
Figure 3: Qualitative comparisons of NFC and NFR for static scenes. Top Row: NeRF, Bottom Row: DVGO.
Static Scene We first empirically study novel view synthesis on common static scenes. We choose the Replica Dataset and Tanks and Temples Advanced(T & as the benchmark datasets. The Replica Dataset encompasses eight complex static scenes characterized by dense geometry, high resolution HDR textures, and sparse training views. T Dataset is popular 3D reconstruction dataset known for its challenging conditions, including weak illumination, uniform appearance surfaces, and large-scale scenes.
We first use DVGO, popular accelerated NeRF variant, as the representative of the NeRF methods because training the accelerated NeRF variants more enviromment--rrende friendly and can significantly reduce the energy costs and carbon emissions of our work. We also evaluate vanilla NeRF and NeuS as two backbones on T & T. Because the vanilla NeRF is still a common baseline in related studies, while NeuS is popular method which combines volume rendering and surface reconstruction.
The quantitative results in Table I and Table support that NFC consistently improves three rep- resentative classes of existing neural field methods. We display the qualitative results in Figure 3 Particularly, DVGO, the accelerated variant which sometimes suffers from reconstructing challenging scenes, can be strongly enhanced, while vanilla NeRF, relatively slow neural field method, can

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative results of DVGO
Table 2: Quantitative results of DVGO, (vanilla) NeRF, NeuS on T&T Dataset. The mean metrics are computed over four scenes of T&T..
Replica Dataset.
Scene
Mode
PSNR(1)
SSIM(1)
LPIPS(()
Regression
13.03
0.508
0.726
Model
Mode
PSNR(1)
SSIM(")
LPIPS(.)
Scene
Classification
34.63
0.934
0.0582
Regression
22.41
0.776
0.236
Socne 2
Regression
14.81
0.654
0.640
DVGO
Classification
0.810
Classification
33.82
0,942
0.0660
NeRF
Regression
22.16
0.679
0.382
Scene 3
Regression
15.66
0.661
0.634
Classificatin
22.68
0.716
0.315
Classfiiatiin
34.04
0.965
0.0451
Regression 19.97
0.620
0.413
Scene
Classification
Regression
36.52
18.17
0.977
0.696
0.0311
0.546
NeuS
Classifiction
0.679
0.317
Regression
15.17
0.504
Scene 5
Classification
35.93
0.974
0.0576
Table 3: Quantitative results of D-NeRF on dy- namic scenes, LEGO and Hook.
Regression
21.33
0.854
0.254
Scene
Classification
0.0994
Scene
Mode
PSNR(1)
SSIM(1)
LPIPS(.)
Regression
22.54
0.865
0.231
Scene
Classfiicatin
34.77
0.966
0.0432
Lego
Regression
21.64
0.839
0.165
Classification
23.11
0.886
0.121
Scene 8
Regression
15.89
0.724
0.519
Classificution
33.40
0.952
0.0775
Hook
Regression
29.25
0,965
0.118
Regression
17.08
0.700
0.507
Classification
29.45
0.967
0.0392
Mean
Classification
34.11
0.956
0.0598
Ground Truth
Regression (Standard)
Classification (Ours)
Dynamic
Sparse
Comrupted
Figure 4: Qualitative comparison of NFC and NFR. Top Row: Dynamic Scenes. Middle Row: Sparse Inputs. Bottom Row: Corrupted Images.
also be consistently enhanced. The image quality improvement of DVGO on Replica Dataset is notably impressive with 99.7% PSNR gain. Given the difficulties in reconstructing complex scenes Replica, such intricate texture patterns in complex objects, varying light condition and sparse training views, the advantages of NFC over NFR become more significant.
Dynamic Scene Dynamic scene rendering is a popular but more challenging task, which requires the capability of neural fields to model time domain. A NeRF variant, D-NeRF, has been specifically designed to render dynamic scenes. We also study how NFC improves D-NeRF on two common dynamic scenes, Lego and Hook (Pumarola al., 2021). The quantitative results in Table 3 demon- strate that D-NeRF-C can render dynamic scenes better. The qualitative results of dynamic scenes in the top row of Figure shows that that D-NeRF-C can produce better quality rendering results than D-NeRF-R, R. Both quantitative and qualitative results demonstrate that D-NeRF-C significantly surpasses D-NeRF-R.

---

Published as a conference paper at ICLR 2024
Challenging Scenes NeRF imposes high requirements on data collection, including relatively dense training views, static illumination conditions, and precise camera calibration. These conditions can be particularly challenging due inherent emrors difficulties in data collection.
While common scenes in NeRF studies often look clear, the training images in practice may be limited corrupted due to realistic difficulties or inherent emror in data collection. For example, collected digital images often contain Gaussian noise due to the sensor-relateds factors [Boytt and Joshi (2015). Robustness data corruption and noise memorization can be important performance metric for neural networks in weakly-superrised learning but rarely touched by previous NeRF studies.
To further understand the effectiveness and robustness of NFC, we empirically study NFC and NFR on two challenging tasks: (1) novel view synthesis with sparse inputs and (2) novel view synthesis with image corruption. Our benchmark for these two tasks the Truck scene the & T Intermediate dataset, with DVGO selected as our backbone due to its adaptability.
Table 4: Quantitative results of neural rendering with sparse training images.
Table 5: Quantitative results of neural rendering with corrupted training images.
Data Size
Mode
PSNR())
SSIM(t)
LPIPS())
20%
Classification
Regression
19.38
14.87
0.530
0,580
Noise Scale
0.2
Regression
Mode
PSNR(1)
21.97
SSIM(1)
0.694
LPIPS(J)
0.406
Regression
18.76
0.637
0.426
Classilication
0.353
40%
Classiliction
21.73
Regression
21.33
0.663
0.451
60%
Classdication
Regression
22.27
21.02
0.728
0.682
0.394
0.329
Classilication
Regression
19.67
0.615
0.391
0.512
Regression
21.72
0.698
0.386
Classification
21.45
0.662
0.429
80%
Classiiiation
First, in the sparse-input task, we train DVGO with the sparse version of a simple Truck scene, called Sparse Truck, where we randomly remove some training images. We visualize the qualitative comparisons of NFC and NFR with sparse inputs in the middle row of Figure 4 The experimental results in Table 4 and Figure T0 (of the appendix) further support that the advantage of NFC over NFR becomes even more significant with fewer training inputs.
Second, in the image-corruption task, inject Gaussian noise with the standard deviation st.d into original Truck images (each color value lies in [0, 1) and obtain the corrupted version, called Corrupted Truck. We visualize the qualitative comparisons of NFC and NFR with corrupted images in Figure 4 The experimental results in Table and Figure [I (of the appendix) further support that the advantage of NFC over NER is robust to image corruption.
NFC can encounter the challenging scenes significantly better than NER with the difficulties in real-world data collection. This makes the proposed NFC even more competitive in the real-world practice. This observation aligns with the significant improve observed for the Replica Dataset, which exactly a group of real-worle challenging scenes.
4.2 NEURAL SURFACE RECONSTRUCTION EXPERIMENTS
Surface reconstruction or geometry reconstruction is fundamental task in both computer vision and computer graphics. Recent neural surface reconstruction methods Yariv et al. (2020) 202T): Oechsle et al. (2021); Wang et al. (2021) represent another class of neural field method. To evaluate the effectiveness of NFC for surface reconstruction, we choose a popular neural surface reconstruction method, NeuS, as the backbone which can reconstruct both RGB images and surface information. The training of NeuS requires no ground-truth -trnt surface information.
Given the requirement for both image and geometric assessments in surface reconstruction tasks, we employ the Replica Dataset, which provides comprehensive visual and geometric ground-truth for both image and geometry evaluation, and the T & T dataset for image quality evaluation. To assess geometric quality, we employ a set of popular geometry quality metrics, including Chamfer-(, Distance, F-score, when ground-truth surface and geometric information is available.
In terms of geometric assessment, our NeuS-C model achieves highly impressive improvements. The quantitative results in Table/ddemonstrre that all geometric metrics show impressive improvements
7

---

Published as a conference paper at ICLR 2024
Ground Truth
Regression
Classffication(Our)))
RGB
Depth
Normal
Figure 5: Qualitative comparisons of RGB rendering, depth rendering and normal rendering between NeuS-R and NeuS-C for neural surface reconstruction. Dataset: Replica Scene (Office 4).
Table 6: Quantitative results of neural surface reconstruction Replica, Model: NeuS.
Scone
Training
PSNRIT:
SSIME1)
LPIPS()
Chandfon
Accraay()
Corpleteest()
PrecidrenT)
Recali()
Fiscorth
Nomel CX")
Regression
25.52
0:443
0.148
18.13
11:60
24.55
17.79
22:20
72.14
Chosciicaiin
6.117
Soas:
Choidicatio
31,37
9.344
0.441
8,522
$.350
3.785
25:27
48.08
47.34
16:25
48.85
75.86
75.16
Resrewize Chocflcatio
28.45
0:577
0.500
40.34
33.50
45.39
0,414
5,600
7.102
67.69
51.76
4
Chacliatio
37.44
8.399
0.8722
53822
5,51
5,212
68.01
72.99
70.40
78.59
S 5
Regrewice Chaclicatin
33.78
0.897
0.221
76.35
11.55
140.3
11.69
33.00
0.000
3.154
70.26
59.05
6
Cbeclication
9.884
$2.21
48.63
73,01
Scasc 7
Regrewize Chocitcation
27.10
0.857
8.915
IU7733
0.18
6.511
22.77
5.400
18:1
6.812
25.33
71.32
21.54
67.44
12:44
66.10
15:26
83.41
73.81
Claietictioon
Meas (Roceeec)
Cloidicatio
Regirsiie
79
33,68
0.77
8.988
0J848
5.165
7,699
10.56
7,430
11.31
41.76
7.957
35.70
36.43
$4.80
11.91
55.56
14.87
76.91
69.77
with NFC across all eight scenes. For example, over the eight surface reconstruction tasks, the mean Chamfer-E, Distance drops by 74.6%, while the mean F-score has improved by 273%. Most surface quality metrics have been improved by more than 10 points. In terms of image quality, NFC also exhibits substantial improvements in all image quality metries over the eight scenes. The qualitative results in Figure rendering. These results quantitatively demonstrate the general effectiveness of NFC in improving neural fields for surface reconstruction.
that NFC significantly improves NeuS in both surface reconstruction and RGB
4.3 DISCUSSION AND ABLATION STUDY
In this section, we further discuss the effectiveness and efficiency of NFC.
Ablation Study We conduct ablation study on Target Encoding and Classification Loss using DVGO and NeuS Replica Dataset in Tables and well Table of the appendix. The quantita- tive results demonstrate that Target Encoding is usually helpful and lead to expected performance improvements, while the Classification Loss term plays a dominant role in the improvements of NFC.
8

---

Published as a conference paper at ICLR 2024
Table 7: Ablation study I. Model: DVGO. Dataset: Replica Scene 6. GPU: A100.
Mode
PSNR(1)
SSIM(1)
LPIPS(()
Training Time(minutes)
Rendering Timo(seconds)
NER (Standard)
21.33
0.854
0.254
9.33
0.312
NFC w/o Target Encoding
27.99
0.935
0.110
9.76
0.312
NFC (Ours)
29.75
0.0994
9,93
0.328
Table Ablation study II. Model: NeuS. Dataset: Replica Scene 7, GPU: A100.
Mode
PSNR(1)
SSIM(1)
LPIPS())
Training Time(hours)
Rendering Time(seconds)
NER (Standard)
27.19
0.867
0.135
2.84
10.27
NFC w/o Target Encoding
30.03
0.899
0.0996
2.92
10.27
NFC. (Ours)
31,28
0.915
0,1773
2.98
10.83
We believe that may be fine enough to employ the classification loss technique alone in some cases, while two components are both useful.
Computational Cost We study the computational costs of NFC and NFR in Tables 7 and g It shows that the extra training cost of NFC is very limited (+6% for DVGO and +4% for NeuS) compared with the significant empirical improvement, and the extra rendering cost is nearly zero.
0.50
2.3
029
Clenit aio-ihi
210
9080 1208 1156e 250c 2000 12705
Figure 6: The curves of PSNR, SSIM, and LPIPS with respect to the hyperparameter NFC robust to a wide range of X. Model: DVGO. Dataset: Replica Scene 3.
Figure 7: The learning curves of NFC and NFR.
Robustness to the hyperparameter À The hyperparameter À controls the weight of the classification loss. We plot how the value of λ affects the model performance in Figure [ The quantitative results shows that wide value range of X can enhance the performance. This suggests that the proposed NFC can be easily employed in practice with the default hyperparameter and limited fine-tuning cost.
Generalization We visualize the training and test PSNR curves of NER and NFC using DVGO on Replica Scene 6 in Figure We observed that NFC has a slightly better training PSNR and a much better test PSNR. This suggests that the main gain of NFC comes from generalization rather than optimization. It is known that generalization closely relate to flatter minima of loss landscape (Hochreiter and Schmidhuber 1997: Xie al. 20211), while neural fields research largely lacks generalization analysis. We leave theoretical interpretations as future work.
Limitations The main limitation is that NFC often remarkably improve generalization. only when poor generalization indeed exist. However, when tasks are simple or neural fields have strong view generalization, the improvements become limited or even negligible (see appendix).
CONCLUSION
In this work, we visited a very fundamental but overlooked topic for neural field methods: regression versus classification. Then we design a NFC framework which can formulate existing neural field methods as classification models rather than regression models. Our extensive experiments support that Target Encoding and classification loss can significantly improve the performance of most existing neural field methods novel view synthesis and geometry reconstruction. Moreover, the improvement of NFC is robust sparse inputs, image noise, and dynamic scenes. While our work mainly focuses on 3D vision and reconstruction, we believe NRC is a general neural field framework. We believe it will be very promising to explore and enhance the generalization of neural fields.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This work was partly supported by the National Natural Science Foundation of China under Grant No. 31771475.
REFERENCES
Akata, Z., Perronnin, F., Harchaoui, Z., and Schmid, C. (2013). Label-embedding for attribute- based classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 819-826.
Akata, Z., Perronnin, F, Harchaoui, Z., and Schmid, C. (2015). Label-embeddingg for image classification. IEEE transactions on pattern analysis and machine intelligence, 38(7):1425- 1438.
Bengio, S., Weston, J., and Grangier, D. (2010). Label embedding trees for large multi-class tasks. Advances in neural information processing systems, 23.
Boyat, A. K. and Joshi, B. K. (2015). A review paper: Noise models in digital image processing. Signal & Image Processing, 6(2):63.
Chen, S. E. and Williams, L. (1993). View interpolation for image synthesis. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques, pages 279-288.
Debevec, P. E., Taylor, C. J., and Malik, J. (1996). Modeling and rendering architecture from photographs: A hybrid geometry-anc image- based approach. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 11 11-20.
Gao, Xu, Q. Su, H., Neumann, U., and Xu, Z. (2023). Strivec: Sparse tri vector radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17569-17579.
Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. (1996). The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 43-54.
Hochreiter, S. and Schmidhuber, J. (1997). Flat minima. Neural Computation, 9(1):1-42.
Jia, B.-B. and Zhang, M.-L. (2021). Multi-dimennional classification via sparse label encoding. In International Conference on Machine Learning, pages 4917-4926. PMLR.
Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015.
Levoy, M. and Hanrahan, P. (1996). Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 31-42.
Liu, L., Gu, J., Zaw Lin, K., Chua, T.-S., and Theobalt, C. (2020). Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33: 33:15611-1566.
Michalkiewicz, M., Pontes, J. K., Jack, D., Baktashmotlagh, M., and Eriksson, A. (2019). Implicit surface representations layers in neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
Mildenhall, B., Srinivasan, P. Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng. R. (2021). Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106.
Niemeyer, M., Mescheder, L., Oechsle, M., and Geiger, A. (2020). Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515.
the
Oechsle, M., Peng, S., and Geiger, A. (2021). Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589-5599.
10

---

Published as a conference paper at ICLR 2024
Osher, S., Fedkiw, R., and Piechor, K. (2004). Level set methods and dynamic implicit surfaces. Appl. Mech. Rev., 57(3):B15-BL B15.
Pumarola, A., Corona, E., Pons-Moll, G., and Moreno-Noguer, F. (2021). D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pages 10318-10327.
Rodríguez, P., Bautista, M. A., Gonzalez, J., and Escalera, S. (2018). Beyond one-hot encoding:
Lower dimensional target embedding. Image and Vision Computing, 75:21-31.
Sabella, (1988). A rendering algorithm for visualizing 3d scalar fields. In Proceedings of the 15th annual conference on Computer graphics and interactive techniques, pages 51-58.
Schlüns, K. and Klette, R. (1997). Local and global integration of discrete vector fields. Springer.
Shum, H. and Kang, S. B. (2000). Review of image-based rendering techniques. In Visual Communi- cations and Image Processing 2000, volume 4067, pages 2-13. SPIE.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. (2020). Implicit neural representations with periodic activation functions. Advances Neural Information Processing Systems, 33:7462-7473.
Sun, C., Sun, M., and Chen, H.-T. (2022). Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469.
Wang, P., Liu, I.., Liu, Y., Theobalt, C., Komura, T., and Wang, W. (2021). Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 34:27171-27182
Wu, G., Yi, T., Fang, J., Xie, L., Zhang. X., Wei, W., Liu, W. Tian, Q. and Wang, X. (2023). 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint 1rXiv:2310.08522
Xie, Y.. Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., Sitzmann, V., and Sridhar, S. (2022a). Neural fields visual computing and beyond. In Computer Graphics Forum, volume 41, pages 641-676. Wiley Online Library.
Xie, Z., He, F., Fu, S., Sato, I., Tao, D., and Sugiyama, M. (2021a). Artificial neural variability for deep learning: On overfitting, noise memorization, and catastrophic forgetting. Neural Computation, 33(8).
Xie, Z., Sato, I., and Sugiyama, M. (2021b). A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations.
Xie, Z., Wang, X., Zhang, H., Sato, I., and Sugiyama, M. (2022b). Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 24459.
Xie, Z., Xu, Z.,, Zhang, J., Sato, I., and Sugiyama, M. (2023a). On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective. In Thirty-sevnnth Conference on Neural Information Processing Systems.
Xie, Z., Yang, X., Yang. Y., Sun, Q. Jiang, Y.. Wang. H., Cai, Y., and Sun, M. (2023b). S3im: Stochastic structural similarity and its unreasonable effectiveness for neural fields. In Thirty- International Conference on Computer Vision.
Xie, 7.,, Yuan, I.., Zhu, 7.,, and Sugiyama, M. (2021c). Positive-negative momentum: Manipulating stochastic gradient noise to improve generalization. In International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11448-11458. PMLR.
Yariy, L., Gu, J., Kasten, Y., and Lipman, Y. (2021). Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815

---

Published as a conference paper at ICLR 2024
Yariv, L., Kasten, Y., Moran, D., Galun, M., Atzmon, M., Ronen, B., and Lipman, Y. (2020).
Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492- 2502..
Yen-Chen, L. (2020).
Nerf-pytorch.
httpss//ggthabb.coo/yeeeeeeeeee
nerf pytorch/
Yu, Z., Chen, A., Antic, B., Peng, S. P., Bhattacharyya, A., Niemeyer, M., Tang, S., Sattler, T., and Geiger, (2022). Sdfstudio: unified framework for surface reconstruction.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In International Conference Machine Learning.
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL SETTINGS AND DETAILS
In this section, we present the experimental settings and details for reproducing the results. The main principle of our experimental setting to fairly compare NFC and NFR for NeRF and the variants. Our experimental settings follows original papers to produce the baselines, unless we specify them otherwise.
Classification Loss Setting We fine-tune the classification loss weight À from 5,,2,5, 10) for the NeRF family and (1, 5, 10, 20, 50, 100) for the NeuS family.
A.1
MODELS AND OPTIMIZATION
DVGO Setting We employ the sourcecode of DVGO (Version 2) in the original project (Sun et al. 2022) without modifying training hyperparameters. So we train DVGO via Adam (Kingma and Ba
2015) with the batch size = 4096. The learning rate of density voxel grids and color/fcature voxel grids is 0.1, and the learning rate of the RGB net (MLP) is 0.001. The total number of iterations is 30000. We multiply the learning rate by 0.1 per 1000 iterations.
NeRF Setting We employ popular open-soure implementation (Yen- 2020) of the original NeRF Again, we follow its defaulted training setting. The learning rate is 0.0005, and the learning rate scheduler is 0.1 |_ter=/5000000
D-NeRF Setting We directly employ the sourcecode of D-NeRF in the original project (Pumarola et al. 202T). The learning rate is 0.0005. The total number of iterations is 800k. The learning rate decay follows the original paper.
NeuS Setting We employ the NeuS implementation of SDFStudio (Yu ct al. 2022) and follow its default hyperparameters. The difference of the hyperparameters between SDFStudio and the original paper (Wang et al. 2021) is that SDFStudio trains 100k iterations, while the original paper trains 300k iterations.
Strivec Setting We directly employ the sourcecode of Strivee in the original project (Gao et al. 2023 and follow its original training setting the classical Chair scene.
4DGS Setting We directly employ the sourcecode of 4DGS eW al. 2023) and follow its original training setting on the classical Mutant scene. As the target encoding is relatively expensive for Gaussian Splatting, we use NFC-4DGS without target encoding as our method.
A.2 DATASETS
Replica Dataset Replica Dataset has no splitted training dataset and test dataset. In the experiments on Replica, if one image index divisible by 10, we move the image to the test dataset; if not, we move the image to the training dataset. Thus, we have 90% images for training and 10% images for evaluation.
T&T Dataset Advanced T&T Dataset Advanced has no splitted training data and test data. We follow the original splitted way in the standard setting. In the experiments on T&T Dataset Advanced. if one image index is divisible by 10, we move the image to the test T&T Dataset Advanced; if not, we move the image to the training dataset. Similarly, we again have 90% images for training and 10% images for evaluation.
T&T Dataset Intermediate T&T Dataset Intermediate has splitted training data and test data. We follow the original splitted way in the standard setting. In the experiments of sparse inputs, we randomly remove the training images. In the experiments corrupted images, we inject Gaussian noise with the scale std into color values of the training images, and clip the corrupted color values into [0, 1].
Dynamic Scenes: LEGO, Hook, and Mutant The three dynamics scenes used in the original D-NeRF paper. We use them in the same way without any modification.
Chair Scene from NeRF Synthetic We also use the classical Chair scene for evaluating NFC and NER with Strivec (Gao et al. 2023). Since the standard Chair scene a quite simple benchmark, we only use 20% training images of Chair to obtain more challenging benchmark in our experiment.
13

---

Published as a conference paper at ICLR 2024
Ground Truth
Regression
Classification Ours)
Figure 8: Qualitative comparisons of Strivec-C and Strivec-R for the classical Chair scene.
B SUPPLEMENTARY EMPIRICAL ANALYSIS AND DISCUSSION
In this section, we present supplementary experimental results.
We present the quantitative results of DVGO, NeRF, and NeuS each T&T scenes in Tables 10 and I respectively.
We present the quantitative results and qualitative results of Trivec over the Chair scene in Table 12 and Figureß; B respectively. We present the qualitative results of 4DGS (Wu et al.) (023) in Figure 9 NFC also improves the performance of Trivec and 4DGS but not that significantly. Perhaps. is because Trivec and 4DGS can generalize well enough for common scenes. We conjecture that the improvement of NFC is relatively significant for those challenging settings.
Table 9:
Quantitative results of DVGO methods on T&T
Seene
Mode
PSNR(T)
SSIM(1)
LPIPS(L)
Regression Classifiation
0.759
0.243
Scene
22.04
0.787
0.192
Scene 2
Classification
Regression
23.96
25.32
0.847
0,875
0.250
0.178
Rogression Classification
18.64
0.697
0.272
Sccne
19.93
0.758
0.197
Scene 4
Regression Classificatin
25.27
0.800
0.179
25.43
0.820
0.146
Mean
Claesificarion
Regression
22.41
23.18
0.810
0.776
0.178
0.236
Sparse inputs We plot the curves of PSNR, SSIM, and LPIPS with respect to the training data size to compare NFC and NFR in Figure 10
Robustness to corrupted images We plot the curves of PSNR, SSIM, and LPIPS with respect to the image noise scale std to compare NFC and NFR in Figure II
Table 10: Quantitative results (vanilla) NeRF on T&T.
Scene
Mode
PSNR(T)
SSIM(1)
LPIPS(()
Scene
Classiication
Regression
22.24
22.20
0.682
0.691
0.342
0.349
Sccne 2
Classifccttio
Regression
23.97
25,01
0.799
0.835
0.289
0.374
Regression
18.63
0.548
0.513
Sccne
Classification
0.369
Regression
23.87
0.688
0.293
Scene
Classiication
Regression
22.17 0.679
0.382
Classification
14

---

Published as a conference paper at ICLR 2024
Table 11: Quantitative results of NeuS on T&:T.
Scene
Mode
PSNR(1)
SSIM(1)
L.PISS())
Regression
0.636
0.393
Scene
Classification
21.31
0.682
0.310
Scene
Classificatin
Regression
22.70
0.794
0.290
Scene
Classification
Regression
17.58
19.08
0.551
0.601
0.428
0.345
Scene 4
Claisification
Regression
22.59
20.32
0.640
0.554
0.322
0.398
Meun
Classification
Regression
21.67
19.97
0.620
0.679
0.413
0.317
Table 12: Quantitative results of Strivec the Chair scene.
PSNR())
SSIM(1)
LPIPS(J)
Training Timethours)
Rendering Time(secoodss)
Regression
28.53
0.944
0.0672
1.16
4.38
Chair
Classification
Ablation study over more scenes We present the ablation study of DVGO over all 8 Replica scenes in Tables 13 The results show that the classification loss plays dominant role in the proposed NFC, while the Target Encoding module still often helpful is most scenes (75%) and lead to expected performance improvements.
Generalization We are surprised by the observation in Figure 7 that the generalization gap becomes the main challenge of neural fields in these experiments. Even if neural field models can fit training data very well, they may fail unexpectedly sometimes. According to existing theoretical knowledge in deep learning, this indicates that loss landscape of neural field models has lot of sharp minima that generalize poorly. This pitfall is possibly mitigated by multiple ways that help selecting flat minima. While this work focuses on reshaping loss landscape, our previous theoretical studies (Xie al.) 2021c; 2022b 2023a) also explore how find flatter minima with advanced training algorithms only. They are designed for general neural networks and likely also work well for neural fields. We tend to believe these studies may also shed lights future directions of theoretically understanding and principally improving (generalization of) neural fields in a plug-and-plee way without modifying the model architectures.
15

---

Published as a conference paper at ICLR 2024
Ground Truth
Regression (Standard)
Classification (Ours)
88
Figure 9: Qualitative comparison of 4DGS-C and 4DGS-R for the dynamic Mutant scene. LIPIS:
0.0167 - 0.009.
Regrustor Cuariitaion
Pugrarier Ceudlkeibn
0.55
0.55
27-
0.45
Argredir
Cloce Franiin
1.
0.6
0.5
0.2
6.0
2.5 2,5 Tranking Size
1.0
0.4
0.2
Dala
Thisro (00h S22
Figure 10: We plot the curves of PSNR, SSIM, and LPIPS with respect to the training data size, namely the portion of training samples kept from the original training dataset. The improvement of NFC is even more significant when the training data size decreases. Model: DVGO. Dataset: T&T-Truck.
Regrisson
calion
Begroaakr
0.5
0.5
0.6
60
12
2.4
2.5
2.5
0.0
02
0.6
0.5
1.0
Figure 11: We plot the curves of PSNR, SSIM, and LPIPS with respect the image noise scale std. The improvement of NFC is more significant when training image is corrupted by random noise. Model: DVGO. Dataset: T&TT-Truck.
16

---

Published as a conference paper at ICLR 2024
Table 13: Ablation study. Model: DVGO. Dataset: Replica 8 scenes.
Scene
Mode
PSNR(1)
SSIM(t)
LPIPS(()
NFR (Standard)
13.03
0,508
0.726
Scene
NFC w/a Target Encoding
32.52
0,929
0.0718
NFC (Ours)
34.63
0.934
0,0582
NFR (Standard)
14.81
0,654
0.640
Scene
NFC w/o Target Encoding
33.92
0.951
0.0615
NFC (Ours)
33.82
0.942
0.0660
NFR (Standard)
15.66
0.661
0.634
Scene
NFC w/o Target Encoding
28.69
0.954
0,0715
NFC (Ours)
34.04
0.965
0.0451
NFR (Standard)
18.17
0.696
0.546
Scene
NFC
Target Encoding
37.35
0.976
0.0329
NPC (Ours)
36.52
0.977
0.0311
NFR (Standard)
15.17
0.640
0.504
Scene
NFC w/o Target Encoding
35.86
0.970
0.0595
NFC (Ours)
35.93
0.974
0.0576
NFR (Standard)
21.33
0.854
0.254
Scene
NFC
Target Encoding
27.99
0.935
0.110
NPC (Ours)
29.75
0,941
0.0994
NFR (Standard)
22.54
0.865
0.231
Scene
NFC w/o Target Encoding
27.66
0,903
0.153
NPC (Ours)
34.77
0,966
0.0432
NFR (Standard)
15.89
0.724
0,519
Scene
NFC w/o Target Encoding
36.11
0,967
0.0638
NFC (Ours)
33.40
0,952
0.0775
NFR (Standard)
17.08
0,700
0,507
Mean
NFC w/o Target Encoding
32.49
0,949
0.0676
NPC (Ours)
34.11
0.956
0.0598
17


Reviews:
[{"review_id": "f9vmR1VUGw", "reviewer": "Reviewer_zn3A", "summary": "The paper proposes a novel Neural Field Classifier (NFC) framework that formulates existing neural field methods as classification tasks rather than regression tasks. The method also uses Target Encoding to encode continuous targets for regression tasks into discrete targets for classification tasks. The overall writing is clear. The experiments cover the comparison between the Regression model and the Classification model.\n\n\nRaise score to 6 after rebuttal.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The overall writing is clear.\n2. The experiments clearly show the improvement of the Classification model over the Regression model.", "weaknesses": "1. Some important reference is missing, such as [a]. \n2. [a] uses the CNN encoders to process the images. The idea of the proposed classifier looks similar to [a].\n3. The biggest problem is the compared method. Among all the experiments, DVGO is compared extensively. However, DVGO is a method designed for acceleration, not for accuracy. It is unfair to compare the accuracy with a method not targeting accuracy.\n4. What is the State-of-the-art accuracy of NeRF? Please compare with these methods regarding accuracy.\n5. What is the computational cost of the proposed classifier-based methods? Please compare with DVGO regarding FLOPs and wall-clock time.\n\n\n[a] pixelNeRF: Neural Radiance Fields from One or Few Images, CVPR 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EuIeohnrB4", "reviewer": "Reviewer_WcH4", "summary": "The paper investigates whether regression can be replaced by classification for neural fields, particularly for the application of novel view synthesis. They propose a framework for transforming regression into classification using a binary encoding. They show that this formulation can effectively be optimized and perform experiments for novel view synthesis, the primary application of neural fields. They benchmark an array of NeRF methods and show that using classification loss improves novel view synthesis metrics by a significant margin. Also they show that classification leads to better robustness to corruption and better sample efficiency.", "questions": "- How many training views are given for the Replica scenes? I couldn't find that detail in the paper. The results for regression in figure 3 look particularly poor. I would expect a crisper result for regression with a reasonable number of views. \n\n- How is sigma being output by the MLP? Is that still being regressed, and if so why not encode it as a classification problem? \n\n- How is the volume rendering part performed? Do you use the standard volume rendering equation and integrate the bits with sigma?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is to my knowledge the first to investigate classification as an objective for neural fields. The idea is interesting and applicable to most neural field architectures and is relatively agnostic to data type.\n\n- The idea is general and could be applied to many different applications. I think this would be of interest to people in the neural field community. \n\n- The results are strong and the experiments are thorough. The paper evaluates on multiple novel view synthesis datasets and benchmarks an array of NeRF variants.\n\nThe figures are illustrative and convey the main ideas of the paper.", "weaknesses": "Overall the weaknesses are relatively minor. \n\n- The experiments are restricted to the domain of novel view synthesis. \n\n- The qualitative results for the baselines look unreasonably poor. I would think that a vanilla NeRF model would look better on the scenes presented in the figures. \n\n- Some parts of the method section are a little unclear. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i59ub8CNAk", "reviewer": "Reviewer_fXmy", "summary": "In this paper, the authors proposed Neural Field Classifier that formulates existing neural field methods as classification tasks with target encoding and a classification loss. The authors conducted extensive experiments and visualizations to show that it is better than the regression based methods.", "questions": "1. Why the proposed classification based formulation perform better than the standard regression based methods?\n2. Why is more robust that regression based methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The formulation for classification with target encoding and BCE loss makes sense and is technically sound to me.\n2. Extensive quantitative and qualitative showed that it is better than standard regression based methods.\n3. Writing is good and easy to follow. The visualizations are also very informative and showed the improvement very clearly.", "weaknesses": "1. Besides the experiments and visualization, there is no theoretical analysis of why the proposed classification based formulation is better than the standard regression methods. It is because that the classification loss is easier to optimize or?\n2. There is also little analysis on why it is more robust that the regression based methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gcNZVt5Tc6", "reviewer": "Reviewer_dKd6", "summary": "This paper proposes to formulate the neural field methods as a classification problem rather than the traditional regression problem. To achieve this, the continuous RGB value is encoded as an 8-bit binary (discrete) vector, and BCE loss is used to train the model. Experiments on novel view synthesis and neural surface reconstruction validate that the proposed classification method is superior to its regression variant.", "questions": "See weaknesses.\n\nNote: Since I'm not an expert in Nerf, I'm truly wondering, why such a simple yet effective formulation is proposed in 2023.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear, and the readability is satisfying to me (not an expert in Nerf);\n\n2. The proposed method is simple yet effective;\n\n3. Experiments on various datasets are conducted to validate the effectiveness of the method.", "weaknesses": "1. The ablation study is insufficient:\n    - It can be observed from Eq. (6) that the original regression loss is still included. Although the authors pointed out that the BCE loss is significantly larger than the regression loss, I'm still curious about the effect of using BCE loss only. Since the BCE loss is more dominant, is the regression loss essential?\n    - It seems that without text encoding, the performance is close to the final version. Moreover, only two scenes of one dataset are used to conduct the ablation study. I'm wondering, on average (all scenes and other datasets), is the text encoding essential, or is it optimal?\n\n2. The implementation and comparison are conducted based on methods published in 2020 and 2021. Only DVGO was published in CVPR 2022, which was also long ago. It is kindly suggested to conduct experiments on more recent SOTA methods to evaluate the effectiveness and generality of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)