Reducing Shape-Radiance Ambiguity in Radiance Fields with a Closed-Form Color Estimation Method
Authors: Qihang Fang
Yafei Song
Keqiang Li
Liefeng Bo
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural radiance field
Novel-view synthesis
Regularization
TL;DR: We reduce the shape-radiance ambiguity in the radiance field with a novel regularization loss on its density field, which is based on our closed-form color estimation method.

Abstract: A neural radiance field (NeRF) enables the synthesis of cutting-edge realistic novel view images of a 3D scene. It includes density and color fields to model the shape and radiance of a scene, respectively. Supervised by the photometric loss in an end-to-end training manner, NeRF inherently suffers from the shape-radiance ambiguity problem, i.e., it can perfectly fit training views but does not guarantee decoupling the two fields correctly. To deal with this issue, existing works have incorporated prior knowledge to provide an independent supervision signal for the density field, including total variation loss, sparsity loss, distortion loss, etc. These losses are based on general assumptions about the density field, e.g., it should be smooth, sparse, or compact, which are not adaptive to a specific scene. In this paper, we propose a more adaptive method to reduce the shape-radiance ambiguity. The key is a rendering method that is only based on the density field. Specifically, we first estimate the color field based on the density field and posed images in a closed form. Then NeRF's rendering process can proceed. We address the problems in estimating the color field, including occlusion and non-uniformly distributed views. Afterwards, it is applied to regularize NeRF's density field. As our regularization is guided by photometric loss, it is more adaptive compared to existing ones. Experimental results show that our method improves the density field of NeRF both qualitatively and quantitatively. Our code is available at https://github.com/qihangGH/Closed-form-color-field.

Reducing Shape-Radiance Ambiguity in Radiance Fields with a Closed-Form Color Estimation Method
Qihang Fang! 12*
Yafei Song*
Keqiang Li
Liefeng Bo
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation,
Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences AAlibaba Group
1,2 (f (fanggihang2000, aii.eiiangg20000ia..a.
3 (huaizzhagg syf, lieffnng..... balbbbaabbaa-aaaaaaaaaaaaaaaaaaa inn-iiin.cccc com
Abstract
A neural radiance field (NeRF) enables the synthesis of cutting-edge realistic novel view images of a 3D scene. It includes density and color fields to model the shape and radiance of scene, respectively. Supervised by the photometric loss in an end-to-end training manner, NeRF inherently suffers from the shape- radiance ambiguity problem, i.e., can perfectly fit training views but does not guarantee decoupling the two fields correctly. To deal with this issue, existing works have incorporated prior knowledge to provide an independent supervision signal for the density field, including total variation loss, sparsity loss, distortion loss, etc. These losses are based on general assumptions about the density field, e.g., it should be smooth, sparse, compact, which are not adaptive scene. In this paper, we propose a more adaptive method reduce the shape- radiance ambiguity. The key is rendering method that is only based on the density field. Specifically, we first estimate the color field based on the density field and posed images in closed form. Then NeRF's rendering process proceed. We address the problems in estimating the color field, including occlusion and non- uniformly distributed views. Afterwards, it is applied to regularize NeRF's density field. As our regularization is guided by photometric loss, it is more adaptive compared to existing ones. Experimental results show that method improves the density field of NeRF both qualitatively and quantitatively. Our code is available at http:///gggtaab.bcm/magaaaagg//a www.sawwb...o/.iiaaaaaaa.a
specific
form- color- field
Introduction
A neural radiance field (NeRF) [20 23 8 32 is cutting-edge technique in computer graphics and computer vision that enables the synthesis of realistic novel view images of 3D scene from collection of posed 2D images. It represents a 3D scene with density and color field, which describe the scene's ss shape and radiance, respectively, Despite its impressive novel-view synthesis capability, the NeRF suffers from inherent shape-radiance ambiguity problem [33] To be more specific, a family of degradation solutions exist, which can perfectly recover training views but produce incorrect shape. As consequence, poor novel views would be rendered.
The shape-radiance ambiguity exists because a baRed method usually renders an image using the Max volume rendering algorithm [17]. It integrates the density and color fields closely. As shown in Fig. Ma), a pixel rendered as the sum of products of density and color values. In general, such a rendering process is supervised by the photometric loss in an end-to-end training manner, where the
Both authors contributed equally to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Density field
Photometric loss
Sparsity loss, TV loss,
Photometric
loss
Pixel value
Density field
Density field
Estimated color field
Pixel value
Color field
(a) NeRF
(b) Volume-based regularization
(c) Our Our regulariizatio
Figure I: Graphical representations of the NeRF and its regularizations.
density and color fields do not receive independent supervision signals. As result, NeRF can either modify its density or color field to reduce the loss. Unfortunately, performing the latter may well fit the training views, but neglect the error in shape.
To reduce the ambiguity, previous work focuses on different aspects. A group of methods utilizes additional assumptions, e.g., sparse or dense depth priors [5] 25
22 29 to bypass the influence of
the color field and directly supervise the density field. group of other methods improves NeRF's representation, e.g., by modeling the surface normal [30], to inherently reduce the ambiguity. Besides, a group of methods incorporates prior belief to the original NeRF's representation for regularization. Typical ones are volume- based loss, such sparsity [0 7 and total variation (TV) loss (26] 77.
27 ray-entropy loss [33, and distortion loss ①.. These losses directly supervise the density field with reasonable prior knowledge, so that the shape-radianed ambiguity can be alleviated. We typically focus on the third group of methods and their effects on explicit NeRF models, which are well-known for their training and rendering efficiency. Despite the efficacy of learning a better density field, most existing regularizations are not aware of the geometry of scene. They are based to be sparse, the density should distribute smoothly in the space, and it should distribute compactly along ray. Thus, such regularizations may be less adaptive too aggressive.
and ray-based loss, such as background entropy loss [S]
general assumptions, e-g., the space tends
In this paper, we propose more adaptive method to regularize the density field, which elegantly breaks the entanglement of the shape and radiance in NeRFs. Specifically, we devise rendering method that only based on the density field as shown Fig. Ic). The key to first estimate the color field given the density field and posed training images by using predefined rules. Then, the NeRF's rendering process can proceed as usual. Notably, the color field is not learnable here, and the rendering results solely depend on the density field. In this case, the photometric loss can well supervise the scene shape. Calculating the color field from a density field, however, faces several hurdles, such as occlusion and non con-uniformly distributed views. We tackle them by weighting the observed colors with transmittance and using a residual color estimation scheme, respectively. Overall, regularization is more adaptive volume and ray- ray-based ones, as it is guided by the photometric loss. The results of two typical explicit NeRFs, including Plenoxels (7] and Direct Voxel Grid Optimization (DVGO) (27 28), the NeRF Synthetic (0)). LLFF 19). and DTU datasets [12 demonstrate the superiority of our method. Overall, our contributions are summarized as follows:
specific scene geometry compared with existing
1. We propose a closed-form color field estimation method given the density field and posed images. We deal with problems that include occlusion and non-uniformly distributed views. Experimental results show that the estimated color field has acceptable rendering quality.
2. We use the estimated color field for regularization, and reduce shape-radiance ambiguity accordingly. Our method is aware of scene shape and so more adaptive. Experimental results show that our loss improves the shape both qualitatively and quantitatively.
3. We implement our regularization method with CUDA kernels. Thus, the computational cost for training a scene is still acceptable.
2 Related Work
Novel-view synthesis. The NeRF 20 has revolutionized the novel-view synthesis technology [4 with its impressive rendering quality. It uses fully-connected neural networks to map 3D location and viewing direction to a density and view-dependent color. Then novel view is synthesized by using the MAX volume rendering algorithm [17]. Besides the implicit neural network representation,

---

explicit NeRFs [4 27 28 21 are proposed to accelerate both training and rendering speed. The density and color field are represented by voxel grids, which store the density values or the color features. In this way, a lightweight multilayer perceptron (MI should suffice for predicting the view-dependent colors. Furthermore, researchers have paid great effort to optimize the rendering quality in challenging scenarios
16 18] training speed [4 2 28 B1, and inference speed [1 B 23 8 32]. Despite these improvements, the NeRF's representation of a scene inherently suffers from the shape-radiance ambiguity problem (33] A complex color field can well fit the training images while leaving an error in geometry. Although Zhang et al. [3] analyzes that the NeRF relieves the problem by using an MLP with limited capacity and low-frequency positional encoding components, the shape-radiance ambiguity problem needs to be solved more systematically.
Regularization methods for the density field. The regularization methods can be divided into colume-based and ray-based ones. The most simple volume-based regularization is the sparsity loss It is based on the fact that most of the space in scene is empty. Another effective regularization is the TV loss [26] 7 which assumes that the density values of adjacent voxels should not change drastically. It helps increase the smoothness of the density field. Ray- regularization methods, on the other hand, impose prior beliefs on how the density along a ray should distribute. forces the final transmittance of ray to be either 1 or O, which means that the ray can only belong to the background or the foreground. This can effectively sharpen the contour of an object. The ray-entropy loss (13) is improved version of the background entropy loss. It argues that the density values along a ray should have a low entropy. This is achieved by minimizing the Shannon entropy of the normalized ray density. Such regularization helps produce better geometry than the vanilla NeRF. The distortion loss [1] shares a similar idea with the ray-entropy loss. It encourages the density along a ray to distribute as compactly as possible. This is achieved by optimizing the non-empty intervals to be narrow and near to each other. Experimental results show that the distortion loss effectively removes the floaters in the free space.
10 32 7
The background entropy loss 053 27
Overall, we notice that the above volume and ray-based methods are all based on general assumptions. They are directly applied to the density field, and may be less adaptive. In this paper, we propose more adaptive method to regularize the density field, which is guided by the photometric loss.
3 Preliminaries
A radiance field is composed of a density field F and a color field Fe, The density F E R of point v R indicates the differential probability of a ray hitting a particle v, and Fc R represents the color emitted by v along the direction d. The NeRF uses volume rendering to render the color (() of a ray r(t) = + td that starts at camera's 's original point following the Max 17 rendering equation
C(r) (t)=// f." T(O)
em (())
Fr dt,
where tn and ty are near and far bounds for the integral, and the T(t) is the transmittance defined as (f-
T(t) = exp
exp
To deal with the integral, the NeRF assumes that the density and color are piece-wise constant along the ray r. Thus, the integrals above become summations.
Given collection of images and their corresponding camera poses. A photometrie loss Cp is applied to minimize the difference between a rendered pixel color ((r) and its ground-truth pixel value ((r) as follows,
Cp
Σ F
((r)) - -(0))
(3)
R
where R a mini-batch of rays.
In such a formulation, let Twe r( = T())Fo Fo a denotes the trassmitanne-densityywviiieee r(c)
color field, the photometric loss in effect optimizes Five to fit training views. An optimal F learned on training views, however, does not guarantee a correct density field, since it can be achieved by infinitely possible combinations of density and color fields. This raises the shape-radianc ambiguity problem.
3

---

Method
The method is divided into two parts. First, claborate the color field estimation method given density field and posed images. Then we use the estimated color field for regularization.
4.1
Closed-Form Color Field Estimation
The key problem in this subsection is to estimate a color field fr, v.d as accurately as possible given a density field F and posed observation images.
Spherical harmonics representation for the color field. We use spherical harmonics (SH) to represent the color of each point. The SH includes a series of orthogonal and complete basis functions defined on the surface of the unit sphere S (9. Following [7 linear combination of SH basis functions is used to approximate the color of point observed from direction d
F v.d Σ Σ
h Y (d).
(4)
E=
where L is the degree of the SH basis, (d) is the SH function with degree L, order and he is its coefficient. Note that he" a function of v. We omit its notation for brevity. Under this formulation, the color estimation problem is equivalent to estimating the SH coefficients.
Similar to Fourier series, the coefficient of specific frequency band can be obtained by integrating the product of the color and the corresponding SH function as follows,
F YE (d)dd des
F (d) pa(d)
des Pa(d) dd
(5)
K F v.d. (ds) dg ~ pa(d).
K pa(d;)
where Pa (d) is the probability density function of directions, and the integral is approximated by Monte Carlo sampling. In our case, we assume that the direction should be uniformly distributed and (d(d) 1. However, cannot randomly select the direction since its color could not be observed. 4n Actually, we only know the samples d, from posed images.
transmittane--weihte color for occlusion handling. The posed images provide set of (ds, Fv.,)) for Eq. (5). We use a colored rectangle as an example scene as shown Fig. 2 where v is point on its surface. Specifically, there are four cameras with centers og, h = 1, 2, 3, is a normalized vector that connects oμ and v. Assume that the pro-
and the direction dę ||o--v||
jection matrix of the htth camera is P&, the v.d; is the observation color at P&v of the image plane. Ideally, if there is no occlusion, the radi- ance emitted by can be directly captured by a camera (Oz and Og). Then, the ra- diance is the same the corresponding observation color (F v.d, and F ). How- ever, it is possible that the point is oc- cluded (og and oj), in which case the ob- servation color (Fy and Fe,d) is not the radiance emitted by v, but the radiance of surface point that is closest to the cam- era center. These observations should be treated differently when estimating the SH coefficients. We resort to the transmittance Ty, & between and Og, i.e.,
Fdd
FF
Observation color
F v.da
03
v.d,
Observation transmittance
Radiance field
Figure 2: Observation colors and transmittance.
Tv = exp (ʃ/
11o 1loo-vil
F
ds) ds
(6)

---

r(; d3) = sdg. As Tv,k exponentially accumulates the density along ray, an occluded ray has very low transmittance. Thus, we use weight factor and modify Eq. (5) follows,
K Tv Tv. Y?" pa(da)
k= Ty. k=1
Residual estimation for non-uniformly distributed views. The estimation given by Eq. strongly affected by non-uniformly distributed sample directions. To calculate the coefficient of frequency band, the integral Eq. (5) relies on the orthogonality the SH basis that
i i= and j m,
Y (d))?((dd 0, otherwise.
(8)
JES
Uniform sampling When we discretize the integral with samples that are non-unifone -onuunffrmlly dis- tributed on the sphere S. Eq. (8) may be violated. For simplicity, we use the product of two trigonometric func-
Nonuniform sampling
tions cos Oz CoS as an example. As shown in Fig. [3 the positive and neg- ative values in uniform samples better compensate each other to approach zero-vane value summation, while the non- uniform case generates large bias.
0.5
2
cos((x))co0(xs) 0.00
cos(Oxx)cos(xz) 0.39
To reduce the estimation bias, we pro- pose residual estimation scheme. The basic idea is to subtract the terms that integrate to zero the integral reduce their influences. Specifically, we further modify Eq. (7) as follows,
Figure 3: Non-uniform samples generate a larger bias.
1
Tv.k "0x Y (dx). Pa(da)
(9)
=k=1 Tv.k k=1
with Cc defined as
"C Fv.d
ig. Yd(d)).
(10)
k=
where m m if L, else m = i. This scheme can effectively reduce estimation biases. Please refer the Appendix A for more explanations.
4.2 Regularization with the Estimated Color Field
We use the estimated color field Fz to reduce the shape-radiance ambiguity, Specifically, we can render a closed-form pixel Cef (r) with fo and FV following the volume rendering equation as per Eq. Then, a closed-form photometric loss (CF loss for short) can calculated as follows,
Lef
Σ ||C(r) (r)(1) FER.,
(11)
ReJ
where Rey is a mini-batch of rays. With weight factor the regularization term ACef added to the training loss for reducing the shape-radiance ambiguity.
Experiments
5.1 Experimental Settings
Datasets. evaluate our method, we conduct experiments on DTU 12]. NeRF synthetic (20)
and LLFF datasets [19]. Each DTU scene has 49 64 images. We evenly select 5 or 6 images
5

---

from them as testing ones. Besides, we downsample the 1600 x 1200 images to be 800 x 600 ones. For NeRF synthetic and LLFF datasets, we follow the conventional train-test split used in previous studies. By default, we use a black background for DTU and NeRF synthetic datasets, as it makes the shape-radiance ambiguity more severe on some scenes.
Baselines. We choose two explicit NeRFs, including Plenoxels 7 and DVGO (27 28), as baselines. Among them, the Plenoxels has been trained with the sparsity loss and TV loss, and DVGO has been trained with the background entropy loss.
Metrics. Besides the commonly applied peak signal-to-noise ratio (PSNR) metric of images, calculate the PSNR of depth for the NeRF Synthetic dataset to evaluate the geometry, since it provides the ground truth depth. Depth values normalized into the range [0, 1] before calculating the PSNR. Additionally, we adopt the metric named inverse mean residual color (IMRC) 66] to evaluate the geometry of scene. It is defined as the raassittance-densityyvweee follows
residual color in dB as
IMRC 10g10 EvEr (1- -v)) (Lēv
(12)
Ever 2B (1- 8x))
where Tv is the transmittance defined in Eq. ( f:v. is the final residual color defined in Eq. (10), and 6y is the half-width of a voxel as the step size. To get deeper insights into Eq. (12). it is a weighted mean squared error of the radiance emitted from all directions dg of all points v. The error or residual color L.vv & of a point v at a direction dg is the difference between the ground truth observation color and recovered radiance based on the geometry. The better geometry should better recover the color field. Thus, the higher the IMRC, the lower the residual color, the better.
Implementation details. The codes of Plenoxels and DVGO are borrowed from the official release. A uniform probability distribution Pa (d) of directions is applied. In other words, Pa(d) = 47 is constant value. Please refer to Appendix [B..I] for more discussions about the probability density function of directions. We implement the CF loss with CUDA kernels. For explicit NeRFs, the 3D space divided into voxel grids. We need estimate the color field for the voxels. Specifically, the block number is set as the number of voxels. Each block has 128 threads and each thread handles a ray that connects the voxel and a camera. Following (7) we set the SH degree to 2, iee. there are 3 (2 + 1)² 27 SH coefficients per voxel. For Plenoxels, the number of rays RR.l is set to 25 for all datasets. The weight factor À is set to 10, 0.1, and 0.5 for the DTU, NeRF synthetic, and LLFF dataset, respectively. For DVGO, the number of rays [Ray is set to 10 for all datasets. The weight factor À is set to 1 and 0.1 for the coarse and fine stage of the DTU dataset, 2 and 0.1 for the coarse and fine stage of the NeRF synthetic dataset, and 0.1 for the LLFF dataset. All the experiments are run on a single NVIDIA Tesla A100 GPU.
5.2 Results of the Closed Form Color Field Estimation Given Trained Density Fields
In this subsection, we verify the effectiveness of the closed-form color estimation. Specifically, we train Plenoxels and DVGO with their original implementations without adding the CF loss. Then, the density fields of these trained models are used to estimate color fields without any training process.
The occlusion handling and residual estimation play significant roles. We conduct an ablation study here, The results based on Plenoxels' density fields are reported in Table T In Fig. 4 we additionally showease an example based on a DVGO's ' density field. Specifically. when we directly estimate SH coefficients according to Eq. 7 without using the residual estimation scheme, the ren- dered images look bad as shown by the first two sub-figures. This because estimating the SH coefficient of one frequency band is affected by other frequency components if we cannot uniformly sample directions from the 2D sphere. Since the images of each DTU scene are approximately taken from half sphere, the views are non-uniformly distributed. As a result, the estimation SH co- efficients suffers from biases. On the other hand, when we estimate without occlusion handling, i.e., without using transmittance to weight an observation, the estimation result is blurred as shown by the third sub- uubffigure. This is because in the estimation, FV.d, of different dy is not guaranteed to be the radiance emitted from the same point v, but possibly that of a point
Table 1: Ablation study on DTU scenes based on Plenoxels' den- sity.
Occlusion handling
Residual PSNR estimation
14.61
13.57
25.99
26.49

---

PSNR: 17.78 w/o both
PSNR: 12.72 W/O residual estimation
PSNR: 27. .9
PSNR: 34.60
PSNR: 37.05
occlusion handling
w/ both
DVGO
Ground truth
Figure 4:
Ablation study on DTU scan 122 based on DVGO's s density.
Trained color field
Closed-form color field
Ground truth
Trained depth
Residual color
DVGO
PSNR: 35.41
PSNR: 33.08
Better geomelry
IMRC: 19.94
Planoxds
PSNR: 34.06
PSNR: 29.09
Worse geometry
IMRC: 17.63
PSNR difference
The worse geometry has larger PSNR drop
(Trained Estimated)
DVGO: 2.33
<
Plenoxels: 4.97
from trained to estimated color fields
PSNR difference
The closed-fone color field is more sensitive to false geometry than the trained one
Trained: 1.35
Estimated: 3.99
(DVGO Plenoxels)
Figure 5: Color field estimation results of DTU scan 63.
nearer to camera. When both occlusion handling and residual estimation are applied, shown in the fourth sub-figure, the rendering result is visually acceptable, and quantitatively. has much higher PSNR that approaches the trained one. To summarize, both the table and figure show that the residual estimation improves the PSNR considerably. With only occlusion handling, the PSNR decreases, but when it is combined with the residual color estimation, the PSNR is further improved.
The estimation results are not only affected by the algorithm, but also by the quality of the density field. Specifically, the estimation is based on an assumption that, if point is not occluded, its radiance along direction is directly determined by the corresponding observation color in the image. This assumption is true for the ideal case, where the density is sharp enough and the color reduces to the surface light field 33 By using this assumption, the mossed-form color field becomes sensitive to false geometry. If the geometry or the density field is not correct or sharp enough, which breaks this assumption, the color estimation results will be bad. Fig. [ we show the estimation results based on the density field of DVGO and Plenoxels, respectively. We also visualize the depth map of scene and render residual color. The Plenoxels suffers sharper PSNR drop from the trained image (34.06) to estimated one (29.09). This is because it has an inferior density field as qualitatively shown by the depth maps. The errors in the density field affect the estimation process and result in poor rendering quality. In contrast, better density field given by the DVGO permits higher PSNR (33.08) of the estimated color field. Note that the closed-form color field is more sensitive to the quality of density field than the trained one, it produces larger PSNR difference (3.99 1.35) between two methods. By using the closed-form photometric loss, the rendering errors caused by an inferior density field can be propaaade to improve the density field.
5.3 Results of Regularization and Comparison
In this subsection, analyze the regularization results. Specifically, we apply regularization to Plenoxels and DVGO and train from scratch. Based on DVGO, we additionally add a fine-tuned distortion loss for comparison. Table 2 reports the PSNR, IMRC, and PSNR of depth.
reports
For Plenoxels, the loss improves all metrics on datasets. It effectively improves the geometry of scene. As shown in Fig. 6 the floaters in the free space (marked by red circles) are penalized,
7

---

Table 2: Comparison of PNSR 1/MMRC 1/(PSNR of depth on the three datasets (Best).
Method
DTU
NeRF synthetic
LLFF
Plenoxels Plenoxels
31.86 / 15.88
29.83 / 16.55 22.70
26.30 / 19.15
+ CF loss
32.08 / 16.66
29.84 / 16.59 23.08
26.40 / 20.02
DVGO
27
28
32.15 / 18.19
31.58 / 17.25 / 25,46
26.23 / 18.87
DVGO
27
28
+ Distortion loss
32.20 / 18.47
31.50 / 17.93 25.68
26.33 / 20.76
DVGO
27
28
CF loss
32.23 / 18.51
32.24/ / 17.61 61/ 26.36
26.26 / 19.66
DVGO
28
Distortion loss
+ CF loss
32.26 18.80
32.23 / 18.90 26.68
26.34 20.92
Plenoxels
+ CF loss
Plenoxels
+ CF loss
Randered image
PSNR: 26.22
PSNR: 26.41
3.36
3354
Renderad depth
Residual color
IMRC: 15.75
IMRC: 16.92
IMRC: 17.13
IMRC: 17.1
Figure 6: Our loss improves the geometry of scene by making the density field sharper and removing the floaters in the free space. The two scenes are DTU scan 37 and 105 from left to right.
and the density field becomes sharper (marked by green circles). This is reasonable because the observation colors of the points in wrong geometry, egg. floaters or a thick surface, tend to be frehufry color variation, and the
very high frequency. The SH of degree 2 fails fit such estimated color field may produce poor rendering results, as represented by high CF loss. During back prokapoppag,tion the CF loss can guide a better density field.
For DVGO, the best results for all metrics on all datasets are obtained when the CF loss is added or both CF loss and distortion loss added. These two losses complement each other's advantages, but the CF loss is more sensitive to specific geometric errors. We visualize the rendered images, depth, and residual color of several scenes for better comparison. As shown in Fig. 7 the DVGO fails to recover two details of a house as marked by blue and red rectangles. The distortion loss relieves the problems, but the windows in the red rectangle are still distorted. We try further increase the weight factor of the distortion loss but this does not help. In contrast, our loss significantly improves the quality of the density field as shown by the depth maps and residual colors. Two more examples are shown in Fig. g It is noteworthy that the scene with mic is trained with abundant views (100 views) uniformly distributed in the upper hemisphere. Under black background, however, severely suffers from the radianee ambiguity problem. A part of the wire is confused with the background, and lots of floaters exist around the wire. The distortion loss makes no difference as it is not aware of the geometry of the scene. The CF loss successfully distinguishes the wire and background, producing a better density field. The right scene Fig. i another example. The DVGO, with or without distortion loss, mistakenly reconstructs the upper surface of the bottom rock. This mistake is avoided by our method. Overall, the CF loss effectively improves the geometry of scene qualitatively and quantitatively.
8

---

DVGO
Distortion loss
+ CF loss
Rerrlesed image
Rendered depth
Residual color
IMRC: 17.45
IMRC: 17.76
IMRC: 1853
Figure 7: Comparison of distortion loss and CF loss on DTU scan 24.
DVGO
+ Distortion loss + CF loss
DVGO
Distortion loss
CF loss
PSNR: 28.41
PSNR: 28.40
PSNR: 34.01
Rendered
image
PSN 2603
PSN MMNS
PSN
Rendered
depli
IMRC: 8.09
IMRC: 8.44
IMRC: 16.29
Restcbal color
IMRC: 17.43
IMRC: 17.95
IMRC: 19.83
Figure 8: Comparison of distortion loss and CF loss on the mic from the NeRF synthetic dataset and
DTU scan 40.
5.4
Computational Cost
In this subsection, we analyze the computational cost. Specifically, in each training step, batch of B rays are used. Then, only the colors of the voxels intersected with these rays need to be estimated. Suppose that each ray involves the order of N voxels, there are M BN voxels. To estimate the color of voxel, the density values along the ray from the voxel to K source cameras need to be calculated. Then the total number of voxels involved is of order KMN KB. In our implementation, each CUDA block handles one of the M voxels, and each thread handles one of the K cameras. As the threads are executed in parallel, in theory, this
Table 3: Training time (minutes) with different batch sizes (B).
DTU NeRF synthetic LLFF
16.0
24.2
32.7
20.5
29.4
25 43.0
25.5
34.6

---

reduces the computation overhead from the order of KBN to N. In practice, we observe that sometimes the computational overhead is affected by the batch size. This is due to the hardware limit of the maximum CUDA threads. When excessive number of threads are used, they are not guaranteed to be executed in parallel. In 3 we report the computational cost of training Plenoxels by using different batch sizes for regularization. The number of source cameras K used are 44 or 58 for the DTU, 100 for NeRF synthetic, and from 17 to 54 for LLFF dataset. The DTU dataset requires longer training time because it is trained for 12 epochs, while the NeRF synthetic dataset is trained for epochs. For 9 epochs, the DTU dataset costs about 24 and 32 minutes for 10 and 25 rays, respectively. Compare these with those of NeRF synthetic dataset, we observe that the large number K 100 cameras does not obviously increase the computational burden.
Conclusion
The shape-radiance ambiguity is significant problem of NeRFs. Even with simple background and abundant training images, wrong geometry can be learned and poor novel-views can be rendered. In this paper, we propose new regularization method to break the entanglement of density and color field. The key is a flosed-form color estimation method, which can recover the color field of scene given density field and a set of posed training images. We overcome the difficulties in estimating the color fields by handling the occlusion and non-uniformly distributed views. The the
image quality rendered by our closed-form color field approaches the trained one. Then, photometric loss derived from the estimated color field to provide an independent supervision signal for the density field. Both the PSNR and IMRC of two explicit NeRFs, including Plenoxels and DVGO, are improved. Experimental results show the capability of the CF loss to correct geometric errors compared with existing volume and ray-based losses.
Ground truth
Trained color field
Closed-for form color field
PSNR 26477
PSNR
Figure 9: Color estimation results of highly reflective object given the trained DVGO density.
Limitations and future work. The closed-form color field has difficulties to perfectly recover highly reflective objects. As shown in Fig. | the surface of the scissors is less reflective compared the ground truth. Although increasing the SH degree can relieve this problem to some extent as shown in Appendix B.1.2 it needs to be solved more systematically. For example, the probability density function pa (d of directions can be better estimated, and a rigorous proof of the bias reduction strategy needs to be further studied. Besides, well-estimated color field has the potential to get rid of the parameterized color field in a NeRF, and then only the density field needs to be trained and stored.
Acknowledgements
This work was supported in part by the National Key Research and Development Program of China under Grant 2021 YFB3301504 in part by the National Natural Science Foundation of China under Grant U19B2029, Grant U1909204, and Grant 92267103, in part by the Guangdong Basic and and in part by the CAS STS
Applied Basic Research Foundation under Grant Dongguan Joint Project under Grant 2021 1600200022.
References
[1] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, IA, USA, June 18-24, 2022, pages 5460-5469. IEEE, 2022.
10

---

[2] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial Radiance Fields. In Computer Vision ECCV 2022 17th European Conference. Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, volume 13692 of Lecture Notes in Computer Science, pages 333-350. Springer, 2022.
[3] Zhiqin Chen, Thomas A. Funkhouser. Peter Hedman, and Andrea Tagliasacchi. MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 16569-16578. IEEE, 2023.
[4] Boyang Deng, Jonathan T. Barron, and Pratul P. Srinivasan. JaxNeRF: An Efficient JAX Implementation of NcRF, 2020.
[5] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised NeRF: Fewer Views and Faster Training for Free. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 12872-12881. IEEE, 2022.
[6] Qihang Fang, Yafci Song, Kegiang Li, Li Shen, Huaiyu Wu, Gang Xiong, and Liefeng Bo. Evaluate Geometry of Radiance Field with Low-frequency Color Prior. arXiv preprint arXiv:2304.0435// 2023.
[7] Sara Fridovich-Keil Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance Fields Without Neural Networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5491-5500. IEEE, 2022.
[8] Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien P. C. Valentin. FastNeRF: High-Fidelity Neural Rendering at 200FPS. In 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 14326-14335, IEEE, 2021.
[9] Robin Green. Spherical Harmonic Lighting: The Gritty Details. In Archives of the game developers conference, volume 56, page 4, 2003.
[10] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul E. Debevec. Baking Neural Radiance Fields for Real-Time View Synthesis. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 5855 3855-5864. IEEE, 2021.
[11] Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. EfficientNeRF Efficient Neural Radiance Fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 12892-12901. IEEE, 2022.
[12] Rasmus Ramsbøl Jensen, Anders Lindbjerg Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large Scale Multi-view Stereopsis Evaluation. 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pages 406-413. IEEE Computer Society, 2014.
[13] Mijeong Kim, Seonguk Seo, and Bohyung Han. InfoNeRF: Ray Entropy Minimization for Few-Shor Neural Volume Rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. New Orleans, LA, USA, June 18-24, 2022, pages 12902-1299 122212911. IEEE, 2022.
[14] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural Sparse Voxel Fields. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12 62, 2020, virtual, 2020.
[15] Stephen Lombardi, Tomas Simon, Jason M. Saragih, Gabriel Schwartz, Andreas M. Lehrmann, and Yaser Sheikh. Neural Volumes: Learning Dynamic Renderable Volumes from Images. ACM Transactions Graphics, 38(4):65:1-65. 2019.

---

[16] Ricardo Brriala, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7210-7219. Computer Vision Foundation / IEEE, 2021.
[17] Nelson L. Max. Optical Models for Direct Volume Rendering, IEEE Transactions on Visualiza- tion and Computer Graphics, 1(2):99-108, 108. 1995.
[18] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualle Pratul P. Srinivasan, and Jonathan T. Barron. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 16169-16178. IEEE, 2022.
[19] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng. and Abhishek Kar. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Transactions on Graphics, 38(4):29:1-29:14
2019.
[20] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes Neural Radiance Fields for View Synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part volume 12346 of Lecture Notes in Computer Science, pages 405 605-421. Springer, 2020.
[21] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding, ACM Transactions on Graphics, 4 11(4):002:1 102:15, 2022.
[22] Malte Prinzler, Otmar Hilliges, and Justus Thies. DINER: Depth-aware Image-based NEural Radiance fields. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 12449-12459, IEEE, 2023.
[23] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 14315-14325, IEEE, 2021.
[24] Gernot Riegler and Vladlen Koltun. Stable View Synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages Computer Vision Foundation / IEEE, 2021.
[25] Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 12882-1288 12891. IEEE, 2022.
[26] Leonid I. Rudin and Stanley J. Osher. Total Variation Based Image Restoration with Free Local Constraints. In Proceedings 1994 International Conference on Image Processing, Austin, Texas, USA, November 13-16, 1994, pages 31-35. IEEE Computer Society, 1994.
[27] Cheng Sun, Min Sun, and Hwann-Tzon Tzong Chen. Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022. New Orleans, LA, USA, June 18-24, 2022, pages 5449-5450 5459. IEEF, 2022.
[28] Cheng Sun, Min Sun, and Hwann- -zznn Chen. Improved Direct Voxel Grid Optimization for Radiance Fields Reconstruction. arXiv preprint arXiv:2206.05083 2022.
[29] Mikaela Angelina Uy, Ricardo Bartin-Brualla, Leonidas Guibas, and Ke Li. SCADE: NeRFs from Space Carving with Ambiguity- Aware Depth Estimates. In IEEE/CVE Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 16527. IEEE, 2023.
12

---

[30] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd E. Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured ftew-Dependen Appearance for Neural Radiance Fields. In IEEE/CVE Conference Computer Vision and Puttern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5481-5490. IEFE, 2022.
[31] Feng Wang, Sinan Tan, Xinghang Li. Zeyue Tian, Yafei Song, and Huaping Liu, Mixed Neural Voxels for Fast Multi-view Video Synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19706-19716, October 2023.
[32] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for Real-time Rendering of Neural Radiance Fields. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17 2021, pages 5732-5741. IEEE, 2021.
[33] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. NeRF++: Analyzing and Improving Neural Radiance Fields. arXiv preprint arXi::2010.07492, 2020.
13

---

Appendix
A More Explanations for the Residual Color Estimation Scheme
We first shed light on why the residual color estimation scheme helps reduce the bias caused by nonuniform sampling. Then we conduct numerical experiments to support our claim. For simplicity, the explanation is based on using Fourier series to approximate one-dimensional signals. The basic idea is the same and can be easily extended to the case that uses spherical harmonics (SH) to approximate the signals defined on a spherical surface.
Suppose that the target signal f(r) defined on [- πं is "good" enough, or satisfies the Dirichlet Conditions, it could be expanded into Fourier series, i...
f(x) = Ao + Σ an cos(nx) + bn sin(næ)] n=1
(13)
Given a set of samples {(12, f(x)))] = I,..,, we aim to estimate the Fourier coefficients up to typical degree kimax. The calculation of the coefficient os well-known
1
["
f(x) cos(kx))x
(14)
1 Ao cos(kx) +
-'o (an cos(nx) cos(kr) b,, sin(nx) cos( (kr) dz.
n=1
Using the Monte Carlo method, the above integral can be approximated by samples Ti,..., 2y as
T
às
2T T
Σ
Ao (ko()) + Σ lan cos(nx;; cos(ke;;) + b,, sin(næ,) cos (kx()))))
(15)
Because of the orthogonal completeness of the trigonometric function set, if uniform sampling is performed and the number of samples is large enough, all other terms in Eq. (15) will approach 0 except for ag cos(kz,) cos(kzz,) Moreover, according the Monte Carlo method, have
2x T
cos(kz) cos(kz) dx
cos(kz,) cos (kare).
(16)
And then,
2T T T
T
āg
R
Σ
ag cos())r,) cos(k);)) = 0g.
2T T
Σ t=1
cos(ker,) cos(kez) z as - = ag-
(17)
In the nonuniform sampling case, however, neg-nggle estimation biases will be generated. As an example, we visualize the bias generated by the direct current component + Et= Ao cos(z,) in Fig, 10 In the uniform sampling case, + Σ Ag cos(n,) is better approaching 0, as the positive (red) and negative (blue) function values are well counteracted. contrast, the nonuniform sampling case induces a larger bias as controlled by the magnitude of Ag- We show such bias in numerical experiment later.
To reduce the bias caused by ET Ao cos());)), we find effective subtract this component in the estimation, i.e.
f(x) cos(kr)ddr
ax
π
I" (f(x) - Au) dor
(18)
π
(an cos(nx) cos(kx) + bn sin(nx)) cos(kax) dx.
π
14

---

Uniform sampling
Nonuniform sampling
Ao
0.5Ag
0
-0.54g
-Ag
-2
2
-2
6
2
Bias:
An cxs(x;)
0.09A
Ao cos(x;)
0.39AA
Figure 10: Nonuniform sampling generates larger estimation bias than that of uniform sampling.
This time the Monte Carlo estimator of a becomes
2 d/.
on cos() cos(kar, + bn sin(nar) cos cos(kr,))
(19)
T
t=1
==1
Compared with åg defined in Eq. Eiz Ag cos(kr,)) Thus, the Monte Carlo estimator
15
0 completely eliminates the bias caused by
2 T
(((a)) Ao) cos(krt)
(20)
derived from 1 {_ - Au) cos (Ka)) )r suffers less bias from nonuniform sampling than
f(z)) cos(kxz)
(21)
derived from 1 J f()) cos(kx)dz. Because Ao is actually unknown, first estimate this direct current component using the Monte Carlo method as
Ao T
Σ f(x)).
(22)
Then, substitute the unknown Ao with Ao in Eq. 20)
Note that, the bias of other components that are integrated to could be analysed and eliminated similarly. We show in the experiments later that such treatment effectively reduces the estimation bias.
As summary, when estimating the coefficient dg of frequency component, we should subtract all other frequency components that are integrated to to reduce the estimation bias as much as possible. Notably, the coefficients for frequency components A: + 1, k + km..x kmaz are unknown we perform the estimation turn from to k. One feasible solution to perform the estimation iteratively. Concretely, in the first round, we only subtract the frequency components 1 to A 1 when estimating dg. Then in the second round, we can subtract all other components. The itcration is ended until the difference of the estimated coefficients in the adjacent round becomes small enough. In the main paper, only one round estimation is performed considering the computational overheads. Notice that because the SH functions also has orthogonal completeness, the above scheme, i.e., estimating the coefficients in turn and eliminating the influences of other frequency components, also makes difference for estimating SH coefficients
have done in the main paper.
We perform numerical experiments about Fourier series support our claim. We compare three estimators, including the original Monte Carlo estimator, Monte Carlo estimator using our bias
15

---

reducing strategy, and the estimator based on the least square method. Specifically, the least square solution sin(a1) ) (in(22)
(
cos(21)
cos ( cos( Kaaw
sin Kmnnnit) sin (2)
((x
cos(22)
I (222)
(23)
(os( (ir()))
sin (maa
f(xT)
where i denotes pseudo-inverse. We make sure that the number of samples is larger than number of coefficients to be estimated.
For the target signals, we randomly choose from trigonometric and polynomial functions, and determine their coefficients randomly. As result, three random target functions are chosen as follows,
[x)) 0.03.,² 2sin(x) + cos(32r,
f2(2) = 10 0.02x + 0.01x² + cos(z) - sin((x)).
(24)
fa(x) = 5 + 0.05I² 0.001 sin(x) 2 cos (2x).
To perform nonuniform sampling, sample 2, from normal distribution A 0²). To compare the three estimators quantitatively, we uniformly sample 1000 points from 2-2a, 2a]. Then, we calculate the root squared error (RMSE) of this 1000 points. Because of the randomness of the estimation process, we repeat the estimation for 100, 000 times and report the mean RMSE (MRMSE) as the final metric, i.e..
100000
1000
MRMSE 100000
Σ j=1
f
(25)
1000
where miy is the j sample in the hth estimation process. f(-) is the ground-truth signal and fe(-) is the estimated signal in the 5th estimation process. The lower the MRMSE, the better.
We set kmas = 3, a 10, and the number of samples for estimation T 10, 20.... 100. The MRMSE results for (ar), fa()), and f1(r) are shown in Fig. 11 It is clear that with our bias reduc- ing strategy, the estimation consistently better than original Monte Carlo estimation. Furthermore, in the case of sparse sampling, the least square method suffers sharp performance degradation. When only 10 samples are available, it is even worse than the original Monte Carlo estimation. When more samples are given, method and the least square method become comparable.
Monte Carlo with bias reducing
Least square
Monte Carlo
f1(x)
10.0
f2(x)
fs(x)
28
40
MRMSEE
5.9-
20
2.00
10.-
1.50
1.25
1.00
Number of samples for coefficient estimation (T)
Figure 11: The MRMSEL results for fi (z). f2(x). and fa(x).
In addition, we add different direct current values to (x to increase its Ag, and fix T 100. The MRMSE results are listed in Table 4 It shows that the estimation bias of Monte Carlo estimation increases with a larger Ag, while our method and the least square method are invariant about such additions.
Finally, the computational cost our method, least square method, and original Monte Carlo method for one estimation with 100 samples are 1.1 10-1 1.052 x 10-1 and 0.978 x 10-1 respectively. The experiments are performed on an Intel i7-9700 CPU. Overall, we can conclude that method effectively reduces estimation biases with acceptable computational overhead.
16

---

Table 4: The MRMSEJ results when adding different direct current values to fi (x).
Addition
5
10
50
100
Monte Carlo with bias reducing
3.89
3.89
3.89
3.89
3,89
3.89
Least square
3.90
3.90
3.90
3.90
3.90
3.90
Monte Carlo
4.09
4.17
4.60
5.31
13.53
25.13
B More Experimental Results
B.1
More Closed-Form Color Estimation Results
B.1.1
Probability Density Functions for Directions
common choice for modeling the distribution of 3D directions is to use the von Mises-Fisher (vMF) distribution defined as follows,
(d:; (4, = 2T G)ec d
(26)
where is the normalized mean direction, and is the concentration parameter. Because the vMF distribution only deals with the unimodal case, we resort to the mixture von Mises-Fisher distribution defined as
pdd: di..... dx, ck = = u(dgi p,e,
(27)
where d1...., dg are known viewing directions as the modes of the distribution. It is noteworthy that when e approaches zero, the probability density function becomes a uniform distribution. Thus, the uniform distribution used in the main paper is special case of the mixture of vMF distribution. The concentration parameter is hyper-parameter. We vary it, and use the probability density function to estimate the color field of DTU scenes given the density field from trained DVGOs. The PSNR results of estimated color fields are reported in Table 5 shows that a large concentration parameter decreases PSNR. Although e = 0.01 is slightly better by 0.01 PSNR, we apply uniform distribution 0) in the main paper for its simplified and more efficient caleulation process.
Table 5: Color estimation results on DTU scenes with different concentration parameters.
Concentration parameter
0
0.01
0.1
PSNR
29.44
29.45
29.43
28.93
27.97
B.1.2 Different SH Degrees
In T report the PSNR of estimated color fields using different SH degrees. It shows clear trend that the PSNR increases with the increase of the SH degree. The computational cost is not obviously affected by the SH degree, but the higher SH degree requires much memory.
Table 6: PSNR of estimated color fields using different SH degrees based on DVGO's density fields.
SH degree
0
1
2
3
SH basis
1
4
9
16
25
256 grid
26.13
26.99
27.89
28,30
28.62
5123 grid
27.22
28.29
29.44
29.98
B.2 More Regularization Results
B.2.1
Visualization and Comparison Ray-based Losses
We train the DVGO 15 27 on a typical scene (DTU scan 40). After training, we calculate common ray-based losses, including the photometric loss, per-point RGB loss (27), background entropy loss
17

---

The existing ray-based losses are not aware about the geometry. False geometry can lead to low loss values.
The CF loss is more sensitive to the false geometry.
PSNR: 31.48
PSNR: 26.42
Ground-truie of training view
Trained color field
Depth
Closed- form color field
4.7%
6.5%
28.0%
Photometric loss
Per-point RGB loss
CF loss
11.0%
11.1%
12.4%
Background entropy loss
Ray-entropy loss
Distortion loss
Depth trained with CF loss
Figure 12: Visualization and comparison of common ray-based losses typical scene with false geometry. All losses are normalized into [0, 1]. The red rectangles mark the false geometry region, and the percentages of the losses in this region the whole image are shown the left-top corners.
15 27,, ray-entropy loss [33, and distortion loss [ of training view based on the trained model. For rayeb losses, each pixel has a loss value, so we can visualize the losses as images as shown in We marked region with a rectangle, where the top surface of the bottom rock is falsely reconstructed. On the left-top corners of the loss images, we list the percentages of the losses in the red rectangle over the whole image. These percentages reflect to what extent the corresponding losses focus on the geometric error. The photometric loss of the training view has very low percentage of 4.7%. This demonstrates that minimizing the photometric loss cannot assure correct geometry, which is caused by the shape-radiance ambiguity problem. For other ray-based losses, they also not sensitive enough to the false geometry, with the largest percentage 12.4%. In contrast, the CF loss has a significantly larger percentage of 28%. It focuses more geometric errors than common ray-based losses. The errors can reflected by the image rendered by the closed-form color field, as shown by the right-top image of Fig. 12 The red rectangle has large closed-form photometric loss, and, more importantly, this loss can be used to improve the geometry of the scene.
Fig. 12
B.2.2 Effect of the Weight Factor À and Number of Rays |Refl
We vary the two typer-parameters. including the weight factor and number of rays, Plenoxels 7 and report the average PSNR and IMRC of the DTU dataset in Table The PSNR and IMRC have an increasing trend when we increase the weight factor. Moreover, in Figs. 13 By increasing the weight factor or the number of rays, the problem of thick surface has been relieved. When the weight factor is too large (A 20), however, the PSNR decreases. This is because the CF loss dominates the training. While it only provides the supervision signals to the density field, the color field cannot be well optimized. Therefore, the weight factor cannot be too large.
showcase an example scene
For the number of rays, increasing it helps improve the geometry of the scene. However, mean higher computational cost. To be more specific, 17 minutes are needed on average to train the DTU scenes without CF loss. When add the CF loss and use 10 rays in every iteration, 32 minutes are needed on average. The time cost further increases to 43 minutes if we use 25 rays. To ensure the training efficiency, do not increase the number of rays further.
rays
18

---

Table 7: Average PSNR1/IMRCT of different weight factors À and numbers of rays |Re on the DTU dataset.
Rey
X =
X
10
10
32.00 16.18
32.02 16.29
32.03
16.44
32.04
16.59
25
32.04 / 16.12
32.06/ / 16.26
32.07 16.50
32.08 16.66
==
2=1
==5
2 = 10
== 20
Rendered
image
PSNR: 27.72
PSNR: 27.79
PSNR: 27. 81
PSNR: 27.79
PSNR: 27.73
Rendered
dlepth
Restdual color
IMRC: 14:96
MRC: 15:46
IMRC: 16.10
IMRC: 16.47
IMRC: 16.73
Figure 13: Effect of the weight factor on DTU scan 24.
|Rayl |Ral 10,2
RRal 25,2 2
RRyl 10,2 10
JRell 25, 10
Rendered image
PSNR: 27.72
PSNR: 27.7
PSNR: 27.79 79
PSNR: 74.74
PSNR: 27.79
Rendered
depth
Residual
color
IMRC: 14:96
IMRC: 15.37
IRRC: 15.46
IMRC: 16:20
IMRC: 16.47
Figure 14: Effect of the number of rays on DTU scan 24.
19


Reviews:
[{"review_id": "jN4DDOIDFc", "reviewer": "Reviewer_qn5t", "summary": "The paper proposes a novel regularization technique for optimizing radiance fields to achieve more accurate geometry alongside realistic novel views. The regularizer is based on a closed-form estimation of spherical harmonic coefficients for view-dependent color, as a function of the training pixel colors and the predicted density field. Using this closed-form procedure to estimate color from density enables regularizing density directly through photometric loss. This regularizer is therefore adaptive to each scene, unlike more general, standard scene-agnostic regularizers like sparsity and total variation. The paper shows empirical improvement in reconstructed geometry via improved rendered depth maps, for two voxel-based scene representations (DVGO and Plenoxels). ", "questions": "My biggest question about this paper is why the proposed method is used as a regularization scheme on top of the usual least squares objective, rather than optimizing a density-only model and directly using the colors from the closed-form predictions. I wouldn’t be surprised if doing so yields even better geometry, and reduces model size as an added bonus. Or if this simpler strategy works worse than what is proposed, do the authors have an explanation why (perhaps related to the limitation illustrated in Figure 9)?\n\nI would also appreciate responses to my comments in the weaknesses section, primarily regarding evaluation weaknesses.", "limitations": "Yes, limitations are discussed to some extent, but the paper would benefit from more discussion of the origin (and potential remedies, if they exist) of the limited quality of the estimated color field.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I really like the observation the paper makes that it is possible to predict color in closed form as a function of density field and training ray colors. It makes a lot of sense that using this closed-form solution for color and then optimizing density directly through photometric loss, should yield more accurate reconstructed density fields. And indeed the results do show improved depth renderings. ", "weaknesses": "presentation weaknesses\n- Figure 2 should include o_k and P_k, and ideally also F_c. In its current form there is a bit of a struggle to match variables in the text to components in the figure.\n- I don’t see much value added by Figure 3. It is used to illustrate the idea that nonuniform or systematically biased sampling can result in a systematic error in the estimated integral. In my view, this idea is clear from textual description, and the space currently dedicated to the figure could be better spent on detail (either textual or ideally graphical) about how the proposed bias correction works.\n- As an added point of confusion around Figure 3, why include a term that is cos(0x) (in both the figure and the accompanying text)? It appears that this is literally referring to cos(0)=1, which has no effect in the example shown.\n- Equation 14 (the definition of the IMRC metric) could benefit from another sentence of explanation, describing the intuition that this metric is an approximation of PSNR in 3D.\n- Figures 4 and 5 would benefit from more substantial captions that explain the experiment being presented and how the results should be interpreted. Currently this information is only available in the text, but it’s best if figures are also self-contained.\n- It’s not clear from the paper whether the results (or which subset of the results) are fine-tuned using pre-trained models vs trained from scratch. I’m not sure if the argument is that fine-tuning with the closed-form loss improves existing pre-trained geometry, or that training from scratch with the closed-form loss as a regularizer produces better geometry, or both.\n- There are scattered typos and minor grammatical issues; please copy-edit the final paper.\nevaluation weaknesses\n- The evaluation uses a black background, whereas in the baseline papers a white background was used. Is there a reason for changing the background color?\n- PSNR values reported for the two baselines (Plenoxels and DVGO) are substantially lower than the values reported in the original papers (29.83 vs 31.71 for Plenoxels, and 31.58 vs 31.95 and 32.8 for DVGO). As far as I’m aware the only difference is the use of the black background; I’m concerned that either this or some other implementation difference makes the baselines used unnecessarily weak.\n- I understand the IMRC metric and see why it’s useful for datasets where ground truth geometry is unavailable, but in my view an even stronger evaluation would be to compare PSNR on depth maps for a dataset where ground truth geometry is available (e.g. this could be rendered for the NeRF-synthetic dataset).\n- Figure 9 shows that there is a gap in quality between renderings produced by the trained color field vs the closed-form color field—can the authors comment on the origin of this gap, and how it might be closed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#  Presentation weaknesses\n\nThank you very much for your detailed comments and suggestions about the presentation. We will try our best to fix these in the final version of the paper.\n\n# Evaluation weaknesses\n\n__W1__: Thanks for your comments. The goal of this work is to study the shape-radiance ambiguity problem of the NeRF model. The black background is a more challenging setting, with which more floaters tend to generate in the free space. It provides a better way to reveal and study the shape-radiance ambiguity problem. To make it clearer, we will add this reason in the final version of the paper in the section of experimental settings. \n\n__W2__: Thanks for your comments. First, we would like to clarify that there are not any other implementation differences except for the use of the black background. The black background is a more challenging setting, so that the PSNR values drop as you mentioned. We would like to emphasize that all the experiments in the paper are done with the same black background images. Thus, the comparison of our method with the baselines is fair under the same challenging setting. Moreover, in practice, photos may be taken at night, which makes their background mainly black, so it is also meaningful to study the model performance under this setting. To make readers clearer, we will explain this in the final version of the paper in the section of experimental results. \n\n__W3__: Thank you very much for providing us a stronger evaluation method. We evaluate the depth maps for the NeRF-synthetic dataset, as the ground truth depth is provided in the test datasets. Because the pixel values of an image for PSNR calculation fall within [0, 1], we first normalize the ground truth and predicted depth to be within this range. Then, we can calculate the PSNR on depth maps. The results are reported in Table below. We can see that for both Plenoxels and DVGO, the CF loss improves the quality of the depth maps.\n\n __Comparison of the PSNR on depth of the NeRF synthetic dataset__\n|Method|Plenoxels|Plenoxels + CF loss|DVGO|DVGO + Distortion loss|DVGO + CF loss|DVGO + Distortion loss + CF loss|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|PSNR on depth|22.70|23.08|25.46|25.68|26.36|26.68|\n\n__W4__: It is difficult to model highly reflective object with low SH degree, especially given few views. The drop in PSNR can be greatly mitigated by using a higher SH degree. Below we report the average PSNR over all DTU scenes from SH degree 0 to 3 (We use degree 2 in the paper). In average, the PSNR increase by 0.54 dB from SH degree 2 to 3.\n\n__Color estimation results with different degrees of SH coefficients on the DTU dataset__\n|SH degree|0|1|2|3|\n|:-:|:-:|:-:|:-:|:-:|\n|PSNR|27.22|28.29|29.44|29.98|\n\n\n# Questions\n\n__Q1__: Thanks a lot for your constructive question. We are also aware that it is possible to optimize a density only model directly using our proposed CF loss. The main difficulty lies on the computational overhead. We give a computational analysis in the answer to the Weakness 3 to the Reviewer hBrk, and report the computational cost in the Table below. A main conclusion is that the computational cost is affected by the batch size or number of rays in practice. In our experiments, we use 25 rays in each batch for regularization in addition to 5000 rays used in the original optimization process, which takes about 43 minutes to train a DTU model in average. \n\n__Mean computational cost using different number of CF rays__\n|Number of CF rays|DTU|NeRF synthetic|LLFF|\n|:-:|:-:|:-:|:-:|\n|0|17 min 43 sec|15 min 59 sec|24 min 11 sec|\n|10|32 min 39 sec|20 min 29 sec|29 min 22 sec|\n|25|43 min 1 sec|25 min 32 sec|34 min 33 sec|\n\nOn one hand, if we increase the number of rays to 5000 for either regularization or optimization for a density only model. The computational cost will be increased from below an hour to a half day. As we focus on the explicit NeRF models in this work, which feature a fast training process, we aim to limit the training time to be within an hour. On the other hand, if we only use 25 rays in each batch to optimize a density only model, the results are worse than the original model that uses 5000 rays in each batch. That is why we apply the CF loss as a regularization.\n\nIt is still interesting to compare the density only model with the original model under a fair setting. Specifically, we train the original Plenoxels also using 25 rays on the DTU dataset, and compare it with the density only model trained by the CF loss also using 25 rays. This is a more challenging setting, as few training rays easily make the model overfit to a local region, and thus exacerbate the shape-radiance ambiguity problem. We report the mean metrics in the Table below. The density only models not only converge, but even yield better PSNR and IMRC.\n\n__Comparison of the Plenoxels and density only model with CF loss by using 25 rays__\n|Method|PSNR|IMRC|\n|:-:|:-:|:-:|\n|Plenoxels|26.12|14.21|\n|Density only model with CF loss|27.82|15.33|\n\nIn summary, we believe that there is great potential to train density only models using the CF loss, but in this work, we use it as a regularization term to meet our goal of acceptable training time. The unique superiority of our regularization method is that even with only the regularization term, the model can converge, which is not achieved by other regularization in NeRF to the best of our knowledge. We will further explore how more CF rays work in the future.\n\n"}, {"review_id": "nF0odNjAmW", "reviewer": "Reviewer_hBrk", "summary": "While nerfs have shown impressive properties in the ability to reconstruct the geometry of three dimensional scenes, there are still some cases where the performances are not as good as expected. This paper focuses on one of these problems, the untangling of the geometry from the color. For that, the authors propose to learn a second color model, using a close form formula based on a spherical harmonics model. On top of the classic ray tracing of nerf, this method requires to trace additional rays that goes from the query point to the cameras. The authors also show the existence of a bias depending on the distribution of view angle and propose a method to correct it.", "questions": "What strategy of point sampling is performed for estimating the color and transmittance of a given point (i.e when the ray from this point to the camera is drawn)?", "limitations": "I don't foresee any potential negative societal impact of the work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper focus on an important problem. Estimating accurately the geometry in complex scene, especially with high specularity, is still a very relevant today.\n\nDespite the heavy computational load, the authors were able to successfully train different models. They also provide results on three different datasets, with two of them being real images.", "weaknesses": "The premise of the method is that density and colors are estimated independently in a nerf model. This is actually false. In particular, in the vanilla nerf model the prediction of the color directly depends on the density. The color MLP takes as input not the coordinates of the point but the features corresponding to the density predicting network. This means that nerf should actually be modeled by (c) in Figure 1 and not (a) as claimed in the paper.\n\nMany variables are not introduced properly making the reading more difficult that it should.\n\nThe method is highly inefficient. For a given ray, for a given sampled point, the method requires to integrate the density along the ray toward each source camera. This means that if the model is trained using $N$ cameras and $M$ points are sampled each time to approximate the integrals, the proposed method requires to estimate the density of $NM^2$ points instead of the usual $M$ points.\n\nThe parameters of the method vary between datasets. No explanation or intuition is provided for that.\n\nRegarding the experiments, the results are quite incremental with an increase of the order of 0.1dB compared to two previous methods. It's also disappointing that the analysis was not performed with more models. We can especially mention tensoRF that also has a model based on SH. It would have also been interesting to compare to methods like regnerf that adds additional regularizers to nerf to be able reconstruct better the geometry even in difficult conditions. Moreover, while the paper focuses on improving the geometry of the trained model, there is no quantitative experiment showing that the proposed method has indeed an impact on the geometry thus learned. Only a few qualitative results are presented with a color map for the depth that is very difficult to interpret. One can also regret the lack of computational analysis. No study are done on the unbiaising process proposed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Weaknesses\n__W1__: Thanks for your comments. The NeRF can be categorized into implicit and explicit models. We focus on the explicit ones in this work. They model the density and color as volumes. The color volume does not receive features from the density one, and so the they can be estimated independently. We believe that the premise of our method is correct.\n\n__W2__: Thanks for your feedback. We will carefully revise the variables.\n\n__W3__: Thanks for your comments. To relieve the computational burden, we implement by CUDA kernels. At each batch of training, a batch of $B$ rays are used. Then, only the colors of the voxels intersected with these rays need to be estimated. Suppose that each ray involves $K$ voxels ($K$ varies for different rays actually), there are $M=BK$ voxels. To estimate the color of a voxel, the density values along the ray from the voxel to $N$ source cameras need to be calculated. Then the total number of voxels involved is of order $NMK=NBK^2$. In our implementation, each CUDA block handles one of the $M$ voxels, and each thread handles one of the $N$ cameras. As the threads are executed in a parallel way, in theory, this reduces the computation overhead from the order of $NBK^2$ to $K$. In practice, we observe that sometimes the computational overhead is affected by the batch size $B$. This is due to the hardware limit of the maximum CUDA threads. When excessive threads are used, they are not guaranteed to be executed parallelly. We report and compare the computational cost below. The computational overhead is acceptable.\n\n__Mean computational cost using different number of CF rays__\n|Number of CF rays|DTU|NeRF synthetic|LLFF|\n|:-:|:-:|:-:|:-:|\n|0|17 min 43 sec|15 min 59 sec|24 min 11 sec|\n|10|32 min 39 sec|20 min 29 sec|29 min 22 sec|\n|25|43 min 1 sec|25 min 32 sec|34 min 33 sec|\n\nThe number of source cameras used are 44 or 58 for DTU, 100 for NeRF synthetic, and from 17 to 54 for LLFF. The DTU requires longer training time because it is trained for 12 epochs, while the NeRF synthetic is trained for 9 epochs. For 9 epochs, the DTU cost about 24 minutes and 32 minutes for 10 and 25 rays, respectively. Compare these with those of NeRF synthetic, we can observe that the large number of 100 cameras does not obviously increase the computational burden. \n\n__W4__: Thanks for your comments. To determine the hyperparameters, we set aside a portion of training data as validation data, and use grid search to find the best hyperperameters. Then, we use all training data for training. Specifically, we try the weight factor for the CF loss including 1, 5, 10, and 20, and the number of rays including 10 and 25.\n\n__W5__: Thanks a lot for your detailed comments. We separate our answer to four sections in the following.\n\n__W5.1 More expriments__\n\nWe follow the default hyperparamter setting in the original work, and train TensoRF, TensoRF with CF loss, NeRF, RegNeRF, and RefNeRF on the DTU dataset. The results are reported below. For the implicit model, the RegNeRF and RefNeRF outperform NeRF due to their regularization and special formulation. Their PSNRs also outperforms those of Plenoxels and DVGO. But they need much longer training time. The explicit model TensoRF performs better, because it allows a high grid resolution due to its decomposition of density and color volumes. By adding CF loss to the TensoRF, the PSNR further improves. This demonstrates the effectiveness of our method.\n\n__More experiments on other models on the DTU dataset__\n|Model|PSNR|IMRC|Training time|\n|:-:|:-:|:-:|:-:|\n|NeRF|31.95|17.95|>10 hours|\n|RegNeRF|32.41|18.90|>1 day|\n|RefNeRF|32.34|18.61|>1 day|\n|TensoRF|32.49|18.86|22 min 56 sec|\n|TesorRF + CF loss|32.66|19.04|57 min 23 sec|\n\n__W5.2 Quantitative results of geometry evaluation__\n\nWe evaluate the geometry of the models trained on the NeRF synthetic dataset by calculating the PSNR on depth maps. The results are reported below. Our CF loss increases the PSNR by 0.38 for the Plenoxels and 0.9 for the DVGO. For the datasets that do not have ground truth depth maps, we use the IMRC metric defined in equation (14) in the manuscript as an alternative. As Reviewer qn5t points out, this metric is an approximation of PSNR in 3D. The wrong geometry, such as floaters in the free space, will be penalized due to the high residual color. Our method increases this metric on both DTU and LLFF dataset. Overall, these quantitative results demonstrate that it makes geometry better.\n\n__Comparison of the PSNR on depth of the NeRF synthetic dataset__\n|Method|Plenoxels|Plenoxels + CF loss|DVGO|DVGO + Distortion loss|DVGO + CF loss|DVGO + Distortion loss + CF loss|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|PSNR on depth|22.70|23.08|25.46|25.68|26.36|26.68|\n\n__W5.3 Computational analysis__\n\nPlease see the answer to weakness (__W3__) you point out.\n\n__W5.4 Study about the unbiasing process__\n\nWe conduct ablation studies for the color estimation process given the density from trained Plenoxels, which involves two components including occlusion handling and residual color estimation or unbiasing. The results are reported below. Please refer to the answer to the weakness 5 (W5) pointed out by the Reviewer LJiP for a detailed discussion. This study demonstrate the effectiveness of the unbiasing process.\n\n__Ablation study of the closed-form color estimation on the DTU dataset__\n|Method|PSNR|\n|:-:|:-:|\n|w/o occlusion handling and residual color estimation|14.61|\n|w/ occlusion and w/o residual color estimation|13.57|\n|w/o occlusion and w/ residual color estimation|25.99|\n|w/ occlusion and residual color estimation|26.49|\n\n\n# Questions\n\n__Q1__: Thanks for your question. The point sampling strategy for the rays that connect this point to a camera center is the same as the one used in original training rays in Plenoxels. Specifically, it uses uniform sampling along the ray that intersects with the density volume with a step size of 0.5 voxel, and no random noise is applied in the sampling."}, {"review_id": "WiX2YVFO5V", "reviewer": "Reviewer_Siz9", "summary": "While NeRFs achieve photorealistic results, they suffer from the shape and radiance ambiguity that is inherent in joint reconstruction. To alleviate this issue, the authors aim to separate the estimations using a closed-form radiance estimation. They leverage Spherical Harmonics to represent the radiance given a density field. With explicit occlusion handling based on the density field, all cameras with visibility for the point can be detected. The colors are then used to fit the SH coefficients for the specific surface point. The authors propose to use this closed-form color to regularize the NeRF training by enforcing similar behavior between the NeRF output and the closed-form color besides the regular photometric loss. \n\nThe effectiveness of this formulation is shown in various experiments and is agnostic to the underlying NeRF method used.", "questions": "- What is the additional overhead of the closed-form color calculation?\n- Would the decrease in performance on highly reflective surfaces be mitigated with more SH bands, or would the method become instable during training?", "limitations": "See the last point in the weakness section.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- One of the most significant advantages of this method is that the regularization is universally applicable to any NeRF method. \n- The method introduces a nice way to add a soft global view-dependent color regularization. \n- Besides issues with highly reflective surfaces, it improves considerably.", "weaknesses": "- Currently, I have the feeling the paper would benefit from an overview figure. This can also be beneficial as a reference in the experiments, where some evaluations only rely on the closed-form color, but in others, it's used as a regularization. Here, one can create a direct link to the overview figure and the symbols for each color output. A simple overview of how the method slots in an existing NeRF framework would be a great addition and can be easily incorporated next to Fig. 1.\n- The authors compared against other ray distribution priors but did not compare with model priors such as RegNeRFs depth/normal input from omnidata.  \n- Similarly, RefNeRF also aims to improve surface reconstruction. How does this method fare against the proposed one?\n- The authors should briefly discuss Riegler et al. - Stable View Synthesis, as it also handles aggregation of features from multiple views. The approach is vastly different, but the rough conceptual idea remains.\nThe authors mentioned that the closed-form color field has issues approximating highly reflective objects, but the limitations are also quite severe. The performance of anything which drastically derivates from Lambertian reflection will suffer. Biasing everything towards Lambertian also solves the shape/radiance ambiguity. So I would like to see if experiments with only lower frequency view encodings or a RefNeRF with an l2 penalty on the estimated roughness are used.\n\nMinor:\n- In 81/82: NeRF’s representation _of_ a scene\n- In 82: remove ‘makes it’: The NeRF’s representation of a scene inherently suffers from shape-radiance ambiguity\n- In 83: error _in_ geometry.\n- In 86: needs to be solved\n- In 174, the authors used minus as a verb. The correct verb is: subtract", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Weaknesses\n\n__W1__: Thanks a lot for your useful suggestion. We will carefully prepare an overview figure that clarifies which part of the experiments is about evaluation and which is about regularization. We will also add links to the symbols, and how the method slots in existing NeRFs.\n\n__W2__ & __W3__: Thanks a lot for you comments. To better compare with RegNeRF and RefNeRF, we additionally train the vanilla NeRF model and report the results below. The hyper-parameters are the same as the deafault values in original implementation, and all the input data are the same as the one we use, which makes it a fair comparison. For the implicit model, the RegNeRF and RefNeRF outperform NeRF due to their regularization and special formulation. Their PSNRs also outperforms those of Plenoxels and DVGO. But they need much longer training time. Specifically, more than one day is needed for training a scene using RegNeRF and RefNeRF, while the explicit models requires only less than an hour. Furthermore, the explicit models also enable faster rendering in the testing phase.\n\n__More experiments on other models on the DTU dataset__\n|Model|PSNR|IMRC|Training time|\n|:-:|:-:|:-:|:-:|\n|NeRF|31.95|17.95|>10 hours|\n|RegNeRF|32.41|18.90|>1 day|\n|RefNeRF|32.34|18.61|>1 day|\n\n__W4__: Thank you very much to provide us the omitted related work. The Stable View Synthesis encodes features of images from source views by using a convolutional network and aggregates the features from these views for predicting the target view. The conceptual idea is roughly the same. We will add further discussion about it to the related work section.\n\nFor the lower frequency view encodings and the issues about the highly reflective objects. Please refer to the answer of __Q2__ below for a detailed discussion.\n\n# Questions\n\n__Q1__: Thanks for your comments. We would like to breifly introduce our implementation first. At each batch of training, a batch of $B$ rays are used. Then, only the SH coefficients of the voxels intersected with these rays need to be estimated. Suppose that each ray involves of $K$ voxels ($K$ varies for different rays actually), there are $M=BK$ voxels. To estimate the SH coefficients of a voxel, the density values along the ray that connects the voxel to each source cameras need to be calculated. Suppose that there are $N$ cameras, then the total number of voxels involved is $NMK=NBK^2$. In our implementation, each CUDA block handles one of the $M$ voxels, and each thread in the block handles one of the $N$ cameras. So each thread handles a ray from a voxel toward each source camera, which involves the calculation of the order of $K$ density values. As the threads are executed in a parallel way, in theory, this reduces the computation overhead from the order of $NBK^2$ to $K$. In practice, we observe that sometimes the computational overhead is affected by the batch size $B$. This is due to the hardware limit of the maximum CUDA threads. When too much threads are used, they are not guaranteed to be executed in a parallel way. We report and compare the computational cost of our method below.\n\n__Mean computational cost using different number of CF rays__\n|Number of CF rays|DTU|NeRF synthetic|LLFF|\n|:-:|:-:|:-:|:-:|\n|0|17 min 43 sec|15 min 59 sec|24 min 11 sec|\n|10|32 min 39 sec|20 min 29 sec|29 min 22 sec|\n|25|43 min 1 sec|25 min 32 sec|34 min 33 sec|\n\nThe number of source cameras used are the number of training images. They are 44 or 58 for DTU, 100 for NeRF synthetic, and from 17 to 54 for LLFF. The DTU requires longer training time because it is trained for 12 epochs, while the NeRF synthetic is trained for 9 epochs. For 9 epochs, the DTU cost about 24 minutes and 32 minutes for 10 and 25 rays, respectively. Compare these with those of NeRF synthetic, we can observe that the large number of 100 cameras does not obviously increase the computational burden. \n\n__Q2__: First of all, we would like to clarify that there are two kinds of results listed in the paper. One is the color estimation results based on the density volume from pre-trained models. The other is the training results of the models armed with the CF loss. We apologize for the confusion and will decribe this more clearly in the final version. The results in Figure 9 is the color estimation results given the density volume from a pre-trained model. It does not involve a training process. In case of the training results, we would like to emphasize that the PSNR for the scene in Figure 9 is improved by using the CF loss. In case of the color estimation process, the PSNR of a closed-form color field does increase by using more SH bands as you have mentioned. Below we report the average PSNR over all DTU scenes from SH degree 0 to 3 (We use degree 2 in the paper).\n\n__Color estimation results with different degrees of SH coefficients on the DTU dataset__\n|SH degree|0|1|2|3|\n|:-:|:-:|:-:|:-:|:-:|\n|PSNR|27.22|28.29|29.44|29.98|\n\nWe also train with lower and higher frequency view encodings, which aim to answers the weakness 4 __W4__ you point out. The lower view encodings deal with non-Lambertian effects worse, and so the PSNR is worse. By training with more SH bands with degee 4, the results become better.\n\n__Traininng reuslts with different degrees of SH coefficients on the DTU dataset__\n|SH degree / SH basis|0 / 1|1 / 4|2 / 9|3 / 16|\n|:-:|:-:|:-:|:-:|:-:|\n|PSNR|31.79|31.95|32.08|32.12|\n|IMRC|16.43|16.56|16.66|16.73|\n|Training time|46 min 31 sec|44 min 42 sec| 43 min 1 sec|44 min 36 sec|\n\n\n\n"}, {"review_id": "IEvWkq4xWw", "reviewer": "Reviewer_LJiP", "summary": "This paper use closed-form spherical harmonics (SH) color-field to regularize the training of NeRF-based models. The SH coefficient of each 3D point has a closed-form solution given a set of observed viewing angles. This work further takes the transmittance from the density field to weigh each observed angle. A residual estimation is proposed to reduce the bias due to non-uniform view angles sampling from the training views. The close-form color-field is only used to regularize the optimization of density field. Experiments show that the proposed regularization can improve quality and alleviate shape-raidance ambiguity.", "questions": "1. I still don't quite convinced why Plenoxel's color field get much worse PSNR when using the proposed closed-form solver comparing to the SGD solver (Fig5,L235-240). As the original Plenoxel use SH as well, I expect the closed-form solver to have a even better or similar MSE (PSNR) under the same fixed density field. Are there any other factors like the degree of SH or the grid resolution affecting the results? More discussions are necessary.\n\n2. Missing experiments for the number of SH degree. The only discussion is Fig9 showing failure cases on high-frequency view-dependent effect. What is the quality improvement and computation cost of using more or less SH degree?\n\n3. This work assumes the view sampling is uniform so the Monte Carlo integrator (Eq.5) has large error when the training viewing angles is not uniform (Fig.3). However, we actually know the camera poses. Is it possible to derive a pdf over viewing angles using the camera poses so that we can have a correct Monte Carlo integrator like: $$\\frac{1}{K} \\sum_{k=1}^K \\frac{f(x_k)}{\\mathrm{pdf}(x_k)} ~.$$", "limitations": "The weakness doesn't discussed the discrepency between the point-wise assumption and the volume rendering (see weakness 3).\n\nIf the authors decide to give up theoratical proof for the proposed bias reduction strategy, it should also be listed as one of the limitation.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Introducing the closed-form SH coefficent solver into NeRF's optimization is an interesting aspect to improve quality of NeRF-based models. I'm interested to see if it is possible to have a closed-form optimal-PSNR color-field in the future given a fixed geometry.", "weaknesses": "1. The manuscript incorrectly use the term \"volumetric-based\" to differentiate voxel-based methods from MLP-based methods (L76). Both of them are volumetric representations mapping spatial coordinates to modality of interest. Please fix it using \"implicit\" and \"explicit\" representations instead.\n\n2. Some baseline results are much worse than their official report. The reported PSNR of Plenoxel on NeRF synthetic dataset is 29.83 which is much lower than their official reported 31.71 in Plenoxel's Table2. The baseline DVGO on Mic scene in Fig8 is much worse than their official results where DVGO doesn't produce the floater artifact around the mic wire. I guess the reason is the background color (from solid white to solid black) which make the shape-radiance ambiguity more severe on some scenes.\nIt is neccessary to have a discussion about the discrepancy of the implementations and the results comparing to the original baselines.\n\n3. The proposed closed-form color-field solution is point-wise without considering the colors from the other points on a ray. As a result, the solved color-field may not be the optimal regarding the photometric loss. This may be one of the reason why Plenoxel (which uses SH as well) have a worse PSNR when using the proposed closed-form solution.\n\n4. Lack of theoratical backup about the bias reduction using residual. The eliminated components (supp's Eq.6) itself is estimated (supp's Eq.10) and is biased due to the non-uniform sampling. It is not clear to me how the overall bias is reduced by using the residual.\n\n5. The ablation study only use a single scene (Fig.4). More examples can make it more convincing.\n\n6. The quantitative improvement is incremental while it needs much more training time (1 hour vs. 10 minutes) and specialized CUDA implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Weaknesses\n\n__W1__: Thank you very much to point out that. We will carefully examine and fix these incorrect terms.\n\n__W2__: Thanks for your comments. The goal of this work is to study the shape-radiance ambiguity problem of the NeRF model. The black background is a more challenging setting, with which more floaters tend to generate in the free space and in some scenes the shape radiance ambiguity problem becoms more severe. That is why the PSNR values drop as you mentioned.\n\nWe would like to emphasize that all experiments are done with the same black background images. Thus, the comparison of our method with the baselines is fair under the same challenging setting. We will explain this in the final version of the paper. \n\n__W3__: Thanks a lot for your thoughtful comments. It is true that the closed-form color-field solution is point-wise. Our assumption is that, if a point is not occluded, or it is on the surface of an object, its color along a direction is directly determined by the corresponding observation color in the image. This assumption is true for the ideal case. It is mentioned in the NeRF++ paper that: ideally, the density should peak at the ground-truth surface location, in which case the color reduces to the surface light field. By using this assumption, the proposed closed-form color-field solution becomes sensitive to false geometry. If the geometry or the density field is not correct or sharp enough, which breaks this assumption, the color estimation results will be worse. This leads to worse rendering colors and so a higher CF loss. We think that it is an advantage instead of weakness, because the worse PSNR or high CF loss in the training will correct the false geometry and lead it to more ideal one through backpropagation. \n\n__W4__: We apologize that we cannot provide a rigorous proof this time. We will try to derive the proof in the future and add this to limitations this time.\n\n__W5__: Thanks for your suggestion. We conduct the ablation study on all scenes of DTU given the density volumes from the trained Plenoxels. Below, we can see that by using the residual color estimation, the PSNR improves considerably. By only add the occlusion handling, the PSNR decreases, but when it is combined with the residual color estimation, the PSNR is further improved. This demonstrates the effectiveness of both occlusion handling and residual color estimation.\n\n__Ablation study results__\n|Method|PSNR|\n|:-:|:-:|\n|w/o occlusion handling and residual color estimation|14.61|\n|w/ occlusion and w/o residual color estimation|13.57|\n|w/o occlusion and w/ residual color estimation|25.99|\n|w/ occlusion and residual color estimation|26.49|\n\n\n__W6__: Thanks for your comments. We additionally calculate the PSNR on depth below for the NeRF synthetic data as it provides ground truth depth. It shows that we achieve appreciable improvement. \n\n__Comparison of the PSNR on depth of the NeRF synthetic dataset__\n|Method|Plenoxels|Plenoxels + CF loss|DVGO|DVGO + Distortion loss|DVGO + CF loss|DVGO + Distortion loss + CF loss|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|PSNR on depth|22.70|23.08|25.46|25.68|26.36|26.68|\n\nWe report the detailed computation cost below. \n\n __Mean computational cost using different number of CF rays__\n|Number of CF rays|DTU|NeRF synthetic|LLFF|\n|:-:|:-:|:-:|:-:|\n|0|17 min 43 sec|15 min 59 sec|24 min 11 sec|\n|10|32 min 39 sec|20 min 29 sec|29 min 22 sec|\n|25|43 min 1 sec|25 min 32 sec|34 min 33 sec|\n\n# Questions\n\n__Q1__: Thanks for your questions. As we explain in the weakness 3 __W3__, the closed-form solver performs worse because it is more sensitive to false geometry due to the point-wise estimation.\n\n__Q2__: We conduct both color estimation and training experiments about SH degrees and report results below. The increase of SH degree does produce better estimation and training results. The computation cost does not vary much, because the SH bands are efficiently handled by CUDA warps. \n\n__Traininng reuslts with different degrees of SH coefficients on the DTU dataset__\n|SH degree / SH basis|0 / 1|1 / 4|2 / 9|3 / 16|\n|:-:|:-:|:-:|:-:|:-:|\n|PSNR|31.79|31.95|32.08|32.12|\n|IMRC|16.43|16.56|16.66|16.73|\n|Training time|46 min 31 sec|44 min 42 sec| 43 min 1 sec|44 min 36 sec|\n\n__Color estimation results with different degrees of SH coefficients on the DTU dataset__\n|SH degree|0|1|2|3|\n|:-:|:-:|:-:|:-:|:-:|\n|PSNR|27.22|28.29|29.44|29.98|\n\n\n__Q3__: Thanks a lot for your constructive question. A common choice for modeling the distribution of 3D directions is to use the von Mises-Fisher (vMF) distribution defined as follows,\n\\begin{equation}\nv(\\mathbf{d}; \\mu, c) = \\frac{c}{2\\pi(e^c - e^{-c})} e^{c \\mu^\\top \\mathbf{d}},\n\\end{equation}\nwhere $\\mu$ is the nomarlized mean direction, and $c$ is the concentration paramter. Because the vMF distribution only deals with the unimodal case. We resort to the mixture of von Mises-Fisher distribution defined as\n\\begin{equation}\np(\\mathbf{d};\\mathbf{d}_1, ..., \\mathbf{d}_K, c) = \\frac{1}{K} \\sum_k  v(\\mathbf{d}; \\mathbf{d}_k, c),\n\\end{equation}\nwhere $\\mathbf{d}_1, ..., \\mathbf{d}_K$ are known viewing directions as the modes of the distribution. It is noteworthy that when $c$ approches zero, the PDF becomes a uniform distribution. So actually the method used in the paper is a special case of the mixture of vMF  distribution. We vary $c$, and use this PDF to estimate the color field given the density field from trained DVGOs. The results are reported below. The PSNR does not increase much, possibly because the viewing directions are few in our study. Despite this, we believe that it is an important improvement and will further study other PDFs and datasets in the future.\n\n__Color estimation results with a modified Monte Carlo integrator on the DTU dataset__\n|Concentration parameter|0|0.01|0.1|1|2|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|PSNR|29.44|29.45|29.43|28.93|27.97|\n\n"}]

Author rebuttal:
Dear all,

We would like to deeply appreciate comments from all reviewers, which are invaluable to improve our paper. We have carefully considered all comments and reponsed in a point-by-point way.

In order to address all concerns and questions raised by reviewers, and make our paper better, we summarize all additional evaluation results based on the reviews and show them as follows. We believe these additional evaluations can help us demonstrate the effectiveness and efficiency of our method.


__Table 1__: Comparison of the Plenoxels and density only model with CF loss by using 25 rays
|Method|PSNR|IMRC|
|:-:|:-:|:-:|
|Plenoxels|26.12|14.21|
|Density only model with CF loss|27.82|15.33|

__Table 2__: Comparison of the PSNR on depth of the NeRF synthetic dataset
|Method|Plenoxels|Plenoxels + CF loss|DVGO|DVGO + Distortion loss|DVGO + CF loss|DVGO + Distortion loss + CF loss|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|PSNR on depth|22.70|23.08|25.46|25.68|26.36|26.68|

__Table 3__: Mean computational cost using different number of CF rays
|Number of CF rays|DTU|NeRF synthetic|LLFF|
|:-:|:-:|:-:|:-:|
|0|17 min 43 sec|15 min 59 sec|24 min 11 sec|
|10|32 min 39 sec|20 min 29 sec|29 min 22 sec|
|25|43 min 1 sec|25 min 32 sec|34 min 33 sec|

__Table 4__: More experiments on other models on the DTU dataset
|Model|PSNR|IMRC|Training time|
|:-:|:-:|:-:|:-:|
|NeRF|31.95|17.95|>10 hours|
|RegNeRF|32.41|18.90|>1 day|
|RefNeRF|32.34|18.61|>1 day|
|TensoRF|32.49|18.86|22 min 56 sec|
|TesorRF + CF loss|32.66|19.04|57 min 23 sec|

__Table 5__: Traininng reuslts with different degrees of SH coefficients on the DTU dataset
|SH degree / SH basis|0 / 1|1 / 4|2 / 9|3 / 16|
|:-:|:-:|:-:|:-:|:-:|
|PSNR|31.79|31.95|32.08|32.12|
|IMRC|16.43|16.56|16.66|16.73|
|Training time|46 min 31 sec|44 min 42 sec| 43 min 1 sec|44 min 36 sec|

__Table 6__: Color estimation results with different degrees of SH coefficients on the DTU dataset
|SH degree|0|1|2|3|
|:-:|:-:|:-:|:-:|:-:|
|PSNR|27.22|28.29|29.44|29.98|

__Table 7__: Color estimation results with a modified Monte Carlo integrator that uses the mixture of von Mises-Fisher distribution on the DTU dataset
|Concentration parameter|0|0.01|0.1|1|2|
|:-:|:-:|:-:|:-:|:-:|:-:|
|PSNR|29.44|29.45|29.43|28.93|27.97|

__Table 8__: Ablation study of the closed-form color estimation on the DTU dataset
|Method|PSNR|
|:-:|:-:|
|w/o occlusion handling and residual color estimation|14.61|
|w/ occlusion and w/o residual color estimation|13.57|
|w/o occlusion and w/ residual color estimation|25.99|
|w/ occlusion and residual color estimation|26.49|





Meta-review: Three reviewers are positive about the paper, in particular, they are excited about the core idea and believe that it will inspire follow-up work. The evaluation missed some baselines and analysis, but the authors' rebuttal provided many of the results. The three reviewers are convinced.

Reviewer hBrk is negative about the paper. The main complaints were the computational overhead, missing experiments with more models like TensoRF, and a slight improvement over the previous methods (given the overhead). 

As the reviews do not converge, the AC has to weigh among different opinions/viewpoints. After reading all the comments, the AC believes that the idea of using closed-form estimation of spherical harmonic coefficients for view-dependent color is quite novel and will be beneficial for the community. Its improvement was indeed not significant but is consistent over various commonly used NeRF models. As discussed in several reviewers' comments, such a work will inspire follow-up work in this area. Considering the pros and cons, the AC decided to recommend acceptance.
Decision: Accept (poster)