Title: Reducing Shape-Radiance Ambiguity in Radiance Fields with a Closed-Form Color Estimation Method
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural radiance field

Novel-view synthesis

Regularization
Abstract: A neural radiance field (NeRF) enables the synthesis of cutting-edge realistic novel view images of a 3D scene. It includes density and color fields to model the shape and radiance of a scene, respectively. Supervised by the photometric loss in an end-to-end training manner, NeRF inherently suffers from the shape-radiance ambiguity problem, i.e., it can perfectly fit training views but does not guarantee decoupling the two fields correctly. To deal with this issue, existing works have incorporated prior knowledge to provide an independent supervision signal for the density field, including total variation loss, sparsity loss, distortion loss, etc. These losses are based on general assumptions about the density field, e.g., it should be smooth, sparse, or compact, which are not adaptive to a specific scene. In this paper, we propose a more adaptive method to reduce the shape-radiance ambiguity. The key is a rendering method that is only based on the density field. Specifically, we first estimate the color field based on the density field and posed images in a closed form. Then NeRF's rendering process can proceed. We address the problems in estimating the color field, including occlusion and non-uniformly distributed views. Afterwards, it is applied to regularize NeRF's density field. As our regularization is guided by photometric loss, it is more adaptive compared to existing ones. Experimental results show that our method improves the density field of NeRF both qualitatively and quantitatively. Our code is available at https://github.com/qihangGH/Closed-form-color-field.

Reducing Shape-Radiance Ambiguity in Radiance Fields with a Closed-Form Color Estimation Method

Abstract

A neural radiance field (NeRF) enables the synthesis of cutting-edge realistic novel view images of a 3D scene. It includes density and color fields to model the shape and radiance of a scene, respectively. Supervised by the photometric loss in an end-to-end training manner, NeRF inherently suffers from the shaperadiance ambiguity problem, i.e., it can perfectly fit training views but does not guarantee decoupling the two fields correctly. To deal with this issue, existing works have incorporated prior knowledge to provide an independent supervision signal for the density field, including total variation loss, sparsity loss, distortion loss, etc . These losses are based on general assumptions about the density field, e.g. , it should be smooth, sparse, or compact, which are not adaptive to a specific scene. In this paper, we propose a more adaptive method to reduce the shaperadiance ambiguity. The key is a rendering method that is only based on the density field. Specifically, we first estimate the color field based on the density field and posed images in a closed form. Then NeRF's rendering process can proceed. We address the problems in estimating the color field, including occlusion and nonuniformly distributed views. Afterwards, it is applied to regularize NeRF's density field. As our regularization is guided by photometric loss, it is more adaptive compared to existing ones. Experimental results show that our method improves the density field of NeRF both qualitatively and quantitatively. Our code is available at https://github.com/qihangGH/Closed-form-color-field .

1 Introduction

A neural radiance field (NeRF) [20, 4, 23, 8, 32] is a cutting-edge technique in computer graphics and computer vision that enables the synthesis of realistic novel view images of a 3D scene from a collection of posed 2D images. It represents a 3D scene with a density and a color field, which describe the scene's shape and radiance, respectively. Despite its impressive novel-view synthesis capability, the NeRF suffers from an inherent shape-radiance ambiguity problem [33]. To be more specific, a family of degradation solutions exist, which can perfectly recover training views but produce incorrect shape. As a consequence, poor novel views would be rendered.

The shape-radiance ambiguity exists because a NeRF-based method usually renders an image using the Max volume rendering algorithm [17]. It integrates the density and color fields closely. As shown in Fig. 1(a), a pixel is rendered as the sum of products of density and color values. In general, such a rendering process is supervised by the photometric loss in an end-to-end training manner, where the

Density field

Color field

Photometric

loss

Density

field

Sparsity loss,

TV loss, …

Density

field

Photometric

loss

Estimated

color field

value

(a) NeRF

(b) Volume-based regularization

(c) Our regularization

density and color fields do not receive independent supervision signals. As a result, NeRF can either modify its density or color field to reduce the loss. Unfortunately, performing the latter may well fit the training views, but neglect the error in shape.

To reduce the ambiguity, previous work focuses on different aspects. A group of methods utilizes additional assumptions, e.g. , sparse or dense depth priors [5, 25, 22, 29], to bypass the influence of the color field and directly supervise the density field. A group of other methods improves NeRF's representation, e.g. , by modeling the surface normal [30], to inherently reduce the ambiguity. Besides, a group of methods incorporates prior belief to the original NeRF's representation for regularization. Typical ones are volume-based loss, such as sparsity [10, 7] and total variation (TV) loss [26, 7], and ray-based loss, such as background entropy loss [15, 27], ray-entropy loss [13], and distortion loss [1]. These losses directly supervise the density field with reasonable prior knowledge, so that the shape-radiance ambiguity can be alleviated. We typically focus on the third group of methods and their effects on explicit NeRF models, which are well-known for their training and rendering efficiency. Despite the efficacy of learning a better density field, most existing regularizations are not aware of the geometry of a scene. They are based on some general assumptions, e.g. , the space tends to be sparse, the density should distribute smoothly in the space, and it should distribute compactly along a ray. Thus, such regularizations may be less adaptive or too aggressive.

In this paper, we propose a more adaptive method to regularize the density field, which elegantly breaks the entanglement of the shape and radiance in NeRFs. Specifically, we devise a rendering method that is only based on the density field as shown in Fig. 1(c). The key is to first estimate the color field given the density field and posed training images by using predefined rules. Then, the NeRF's rendering process can proceed as usual. Notably, the color field is not learnable here, and so the rendering results solely depend on the density field. In this case, the photometric loss can well supervise the scene shape. Calculating the color field from a density field, however, faces several hurdles, such as occlusion and non-uniformly distributed views. We tackle them by weighting the observed colors with transmittance and using a residual color estimation scheme, respectively. Overall, our regularization is more adaptive to specific scene geometry compared with existing volume and ray-based ones, as it is guided by the photometric loss. The results of two typical explicit NeRFs, including Plenoxels [7] and Direct Voxel Grid Optimization (DVGO) [27, 28], on the NeRF Synthetic [20], LLFF [19], and DTU datasets [12] demonstrate the superiority of our method. Overall, our contributions are summarized as follows:

We propose a closed-form color field estimation method given the density field and posed images. We deal with problems that include occlusion and non-uniformly distributed views. Experimental results show that the estimated color field has acceptable rendering quality.

We use the estimated color field for regularization, and reduce shape-radiance ambiguity accordingly. Our method is aware of scene shape and so more adaptive. Experimental results show that our loss improves the shape both qualitatively and quantitatively.

We implement our regularization method with CUDA kernels. Thus, the computational cost for training a scene is still acceptable.

2 Related Work

Novel-view synthesis . The NeRF [20] has revolutionized the novel-view synthesis technology [24] with its impressive rendering quality. It uses fully-connected neural networks to map a 3D location and viewing direction to a density and view-dependent color. Then a novel view is synthesized by using the MAX volume rendering algorithm [17]. Besides the implicit neural network representation,

explicit NeRFs [14, 7, 27, 28, 21, 2] are proposed to accelerate both training and rendering speed. The density and color field are represented by voxel grids, which store the density values or the color features. In this way, a lightweight multilayer perceptron (MLP) should suffice for predicting the view-dependent colors. Furthermore, researchers have paid great effort to optimize the rendering quality in challenging scenarios [1, 16, 18], training speed [14, 7, 27, 28, 21, 31], and inference speed [11, 10, 3, 23, 8, 32]. Despite these improvements, the NeRF's representation of a scene inherently suffers from the shape-radiance ambiguity problem [33]. A complex color field can well fit the training images while leaving an error in geometry. Although Zhang et al. [33] analyzes that the NeRF relieves the problem by using an MLP with limited capacity and low-frequency positional encoding components, the shape-radiance ambiguity problem needs to be solved more systematically.

Regularization methods for the density field . The regularization methods can be divided into volume-based and ray-based ones. The most simple volume-based regularization is the sparsity loss [10, 32, 7]. It is based on the fact that most of the space in a scene is empty. Another effective regularization is the TV loss [26, 7], which assumes that the density values of adjacent voxels should not change drastically. It helps increase the smoothness of the density field. Ray-based regularization methods, on the other hand, impose prior beliefs on how the density along a ray should distribute. The background entropy loss [15, 27] forces the final transmittance of a ray to be either 1 or 0 , which means that the ray can only belong to the background or the foreground. This can effectively sharpen the contour of an object. The ray-entropy loss [13] is an improved version of the background entropy loss. It argues that the density values along a ray should have a low entropy. This is achieved by minimizing the Shannon entropy of the normalized ray density. Such a regularization helps produce better geometry than the vanilla NeRF. The distortion loss [1] shares a similar idea with the ray-entropy loss. It encourages the density along a ray to distribute as compactly as possible. This is achieved by optimizing the non-empty intervals to be narrow and near to each other. Experimental results show that the distortion loss effectively removes the floaters in the free space.

Overall, we notice that the above volume and ray-based methods are all based on general assumptions. They are directly applied to the density field, and so may be less adaptive. In this paper, we propose a more adaptive method to regularize the density field, which is guided by the photometric loss.

3 Preliminaries

A radiance field is composed of a density field F σ and a color field F c . The density F σ v ∈ R of a point v ∈ R 3 indicates the differential probability of a ray hitting a particle at v , and F c v , d ∈ R 3 represents the color emitted by v along the direction d . The NeRF uses volume rendering to render the color ˆ C ( r ) of a ray r ( t ) = o + t d that starts at a camera's original point o following the Max [17] rendering equation

where t n and t f are near and far bounds for the integral, and the T ( t ) is the transmittance defined as

To deal with the integral, the NeRF assumes that the density and color are piece-wise constant along the ray r . Thus, the integrals above become summations.

Given a collection of images and their corresponding camera poses. A photometric loss L p is applied to minimize the difference between a rendered pixel color ˆ C ( r ) and its ground-truth pixel value C ( r ) as follows,

where R is a mini-batch of rays.

In such a formulation, let F wc r ( t ) , d = T ( t ) F σ r ( t ) F c r ( t ) , d denotes the transmittance-density-weighted color field, the photometric loss in effect optimizes F wc r ( t ) , d to fit training views. An optimal F wc r ( t ) , d learned on training views, however, does not guarantee a correct density field, since it can be achieved by infinitely possible combinations of density and color fields. This raises the shape-radiance ambiguity problem.

4 Method

The method is divided into two parts. First, we elaborate on the color field estimation method given a density field and posed images. Then we use the estimated color field for regularization.

4.1 Closed-Form Color Field Estimation

The key problem in this subsection is to estimate a color field ˆ F c v , d as accurately as possible given a density field F σ v and posed observation images.

Spherical harmonics representation for the color field . We use spherical harmonics (SH) to represent the color of each point. The SH includes a series of orthogonal and complete basis functions defined on the surface of the unit sphere S [9]. Following [7], a linear combination of SH basis functions is used to approximate the color of a point v observed from a direction d as

where L is the degree of the SH basis, Y ℓ m ( d ) is the SH function with a degree ℓ , order m , and h m ℓ is its coefficient. Note that h m ℓ is a function of v . We omit its notation for brevity. Under this formulation, the color estimation problem is equivalent to estimating the SH coefficients.

Similar to Fourier series, the coefficient of a specific frequency band can be obtained by integrating the product of the color and the corresponding SH function as follows,

where p d ( d ) is the probability density function of directions, and the integral is approximated by Monte Carlo sampling. In our case, we assume that the direction should be uniformly distributed and p d ( d ) ≡ 1 4 π . However, we cannot randomly select the direction since its color could not be observed. Actually, we only know the samples d k from posed images.

Transmittance-weighted color for occlusion handling . The posed images provide a set of { d k , F c v , d k } for Eq. (5). We use a colored rectangle as an example scene as shown in Fig. 2, where v is a point on its surface. Specifically, there are four cameras with centers o k , k = 1 , 2 , 3 , 4 , and the direction d k = o k -v ∥ o k -v ∥ is a normalized vector that connects o k and v . Assume that the projection matrix of the k th camera is P k , the F c v , d k is the observation color at P k v of the image plane.

Ideally, if there is no occlusion, the radiance emitted by v can be directly captured by a camera ( o 2 and o 3 ). Then, the radiance is the same as the corresponding observation color ( F c v , d 2 and F c v , d 3 ). However, it is possible that the point v is occluded ( o 0 and o 1 ), in which case the observation color ( F c v , d 0 and F c v , d 1 ) is not the radiance emitted by v , but the radiance of a surface point that is closest to the camera center. These observations should be treated differently when estimating the SH coefficients. We resort to the transmittance T v ,k between v and o k , i.e. ,

v

𝐨0

𝐨2

𝐨3

Observation

color

Observation

transmittance

𝐨1

Radiance field

ℱ𝐯,𝐝0

𝑐

ℱ𝐯,𝐝1

𝑐

ℱ𝐯,𝐝2

𝑐

𝑐

where r ( s ; v , d k ) = v + s d k . As T v ,k exponentially accumulates the density along a ray, an occluded ray has a very low transmittance. Thus, we use it as a weight factor and modify Eq. (5) as follows,

Residual estimation for non-uniformly distributed views . The estimation given by Eq. (7) is strongly affected by non-uniformly distributed sample directions. To calculate the coefficient of a frequency band, the integral in Eq. (5) relies on the orthogonality of the SH basis that

When we discretize the integral with samples that are non-uniformly distributed on the sphere S , Eq. (8) may be violated. For simplicity, we use the product of two trigonometric functions cos 0 x · cos x as an example. As shown in Fig. 3, the positive and negative values in uniform samples better compensate each other to approach a zero-value summation, while the nonuniform case generates a large bias.

To reduce the estimation bias, we propose a residual estimation scheme. The basic idea is to subtract the terms that integrate to zero in the integral to reduce their influences. Specifically, we further modify Eq. (7) as follows,

-1

1

0

0.5

-0.5

Uniform sampling

Nonuniform sampling

-2

0

2

-2

0

2

1

𝐾 ෍

𝑘=1

𝐾

cos(0𝑥𝑘 )cos(𝑥𝑘 ) = -0.09

1

𝐾 ෍

𝑘=1

𝐾

cos(0𝑥𝑘)cos(𝑥𝑘 ) = 0.39

where ˜ m = m if i = ℓ , else ˜ m = i . This scheme can effectively reduce estimation biases. Please refer to the Appendix A for more explanations.

4.2 Regularization with the Estimated Color Field

We use the estimated color field ˆ F c v , d to reduce the shape-radiance ambiguity. Specifically, we can render a closed-form pixel C cf ( r ) with ˆ F c v , d and F σ v following the volume rendering equation as per Eq. (1). Then, a closed-form photometric loss (CF loss for short) can be calculated as follows,

where R cf is a mini-batch of rays. With a weight factor λ , the regularization term λ L cf is added to the training loss for reducing the shape-radiance ambiguity.

5 Experiments

5.1 Experimental Settings

Datasets . To evaluate our method, we conduct experiments on DTU [12], NeRF synthetic [20], and LLFF datasets [19]. Each DTU scene has 49 or 64 images. We evenly select 5 or 6 images with ˜ c m ℓ k defined as

from them as testing ones. Besides, we downsample the 1600 × 1200 images to be 800 × 600 ones. For NeRF synthetic and LLFF datasets, we follow the conventional train-test split used in previous studies. By default, we use a black background for DTU and NeRF synthetic datasets, as it makes the shape-radiance ambiguity more severe on some scenes.

Baselines . We choose two explicit NeRFs, including Plenoxels [7] and DVGO [27, 28], as baselines. Among them, the Plenoxels has been trained with the sparsity loss and TV loss, and DVGO has been trained with the background entropy loss.

Metrics . Besides the commonly applied peak signal-to-noise ratio (PSNR) metric of images, we calculate the PSNR of depth for the NeRF Synthetic dataset to evaluate the geometry, since it provides the ground truth depth. Depth values are normalized into the range [0 , 1] before calculating the PSNR. Additionally, we adopt the metric named inverse mean residual color (IMRC) [6] to evaluate the geometry of a scene. It is defined as the transmittance-density-weighted residual color in dB as follows

where T v ,k is the transmittance defined in Eq. (6), ˜ c L L v ,k is the final residual color defined in Eq. (10), and δ v is the half-width of a voxel as the step size. To get deeper insights into Eq. (12), it is a weighted mean squared error of the radiance emitted from all directions d k of all points v . The error or residual color ˜ c L L v ,k of a point v at a direction d k is the difference between the ground truth observation color and recovered radiance based on the geometry. The better geometry should better recover the color field. Thus, the higher the IMRC, the lower the residual color, the better.

Implementation details . The codes of Plenoxels and DVGO are borrowed from the official release. A uniform probability distribution p d ( d ) of directions is applied. In other words, p d ( d ) ≡ 1 4 π is a constant value. Please refer to Appendix B.1.1 for more discussions about the probability density function of directions. We implement the CF loss with CUDA kernels. For explicit NeRFs, the 3D space is divided into voxel grids. We need to estimate the color field for the voxels. Specifically, the block number is set as the number of voxels. Each block has 128 threads and each thread handles a ray that connects the voxel and a camera. Following [7], we set the SH degree to 2 , i.e. , there are 3 ∗ (2 + 1) 2 = 27 SH coefficients per voxel. For Plenoxels, the number of rays |R cf | is set to 25 for all datasets. The weight factor λ is set to 10 , 0 . 1 , and 0 . 5 for the DTU, NeRF synthetic, and LLFF dataset, respectively. For DVGO, the number of rays |R cf | is set to 10 for all datasets. The weight factor λ is set to 1 and 0 . 1 for the coarse and fine stage of the DTU dataset, 2 and 0 . 1 for the coarse and fine stage of the NeRF synthetic dataset, and 0 . 1 for the LLFF dataset. All the experiments are run on a single NVIDIA Tesla A100 GPU.

5.2 Results of the Closed-Form Color Field Estimation Given Trained Density Fields

In this subsection, we verify the effectiveness of the closed-form color estimation. Specifically, we train Plenoxels and DVGO with their original implementations without adding the CF loss. Then, the density fields of these trained models are used to estimate color fields without any training process.

The occlusion handling and residual estimation play significant roles. We conduct an ablation study here. The results based on Plenoxels' density fields are reported in Table 1. In Fig. 4, we additionally showcase an example based on a DVGO's density field. Specifically, when we directly estimate SH coefficients according to Eq. (7) without using the residual estimation scheme, the rendered images look bad as shown by the first two sub-figures. This is because estimating the SH coefficient of one frequency band is affected by other frequency components if we cannot uniformly sample directions from the 2D sphere. Since the images of each DTU scene are approximately taken from a half sphere, the views are non-uniformly distributed. As a result, the estimation of SH coefficients suffers from biases. On the other hand, when we estimate

Table 1: Ablation study on DTU scenes based on Plenoxels' den- sity.

without occlusion handling, i.e. , without using transmittance to weight an observation, the estimation result is blurred as shown by the third sub-figure. This is because in the estimation, F c v , d k of different d k is not guaranteed to be the radiance emitted from the same point v , but possibly that of a point

PSNR: 37.05

PSNR: 34.60

PSNR: 17.72

PSNR: 17.78

w/o both

w/o residual

estimation

w/ both

DVGO

Ground truth

w/o occlusion

handling

PSNR: 27.99

Ground truth

DVGO

Closed-form

color field

PSNR: 35.41

PSNR: 33.08

Trained

color field

Trained depth

Plenoxels

PSNR: 34.06

PSNR: 29.09

Residual color

IMRC: 19.94

IMRC: 17.63

Better geometry

Worse geometry

PSNR difference

(DVGO - Plenoxels)

The closed-form color field is more sensitive

to false geometry than the trained one

Trained: 1.35

Estimated: 3.99

<

PSNR difference

(Trained - Estimated)

The worse geometry has a larger PSNR drop

from trained to estimated color fields

DVGO: 2.33

Plenoxels: 4.97

<

:

:

nearer to a camera. When both occlusion handling and residual estimation are applied, as shown in the fourth sub-figure, the rendering result is visually acceptable, and quantitatively, has a much higher PSNR that approaches the trained one. To summarize, both the table and figure show that the residual estimation improves the PSNR considerably. With only occlusion handling, the PSNR decreases, but when it is combined with the residual color estimation, the PSNR is further improved.

The estimation results are not only affected by the algorithm, but also by the quality of the density field. Specifically, the estimation is based on an assumption that, if a point is not occluded, its radiance along a direction is directly determined by the corresponding observation color in the image. This assumption is true for the ideal case, where the density is sharp enough and the color reduces to the surface light field [33]. By using this assumption, the closed-form color field becomes sensitive to false geometry. If the geometry or the density field is not correct or sharp enough, which breaks this assumption, the color estimation results will be bad. In Fig. 5, we show the estimation results based on the density field of DVGO and Plenoxels, respectively. We also visualize the depth map of the scene and render the residual color. The Plenoxels suffers a sharper PSNR drop from the trained image ( 34 . 06 ) to estimated one ( 29 . 09 ). This is because it has an inferior density field as qualitatively shown by the depth maps. The errors in the density field affect the estimation process and result in poor rendering quality. In contrast, a better density field given by the DVGO permits a higher PSNR ( 33 . 08 ) of the estimated color field. Note that the closed-form color field is more sensitive to the quality of density field than the trained one, as it produces a larger PSNR difference ( 3 . 99 > 1 . 35 ) between two methods. By using the closed-form photometric loss, the rendering errors caused by an inferior density field can be back-propagated to improve the density field.

5.3 Results of Regularization and Comparison

In this subsection, we analyze the regularization results. Specifically, we apply our regularization to Plenoxels and DVGO and train from scratch. Based on DVGO, we additionally add a fine-tuned distortion loss for comparison. Table 2 reports the PSNR, IMRC, and PSNR of depth.

For Plenoxels, the CF loss improves all metrics on all datasets. It effectively improves the geometry of a scene. As shown in Fig. 6, the floaters in the free space (marked by red circles) are penalized,

IMRC: 15.75

PSNR: 26.22

IMRC: 16.92

PSNR: 26.41

Rendered image

Residual color

Rendered depth

IMRC: 17.13

PSNR: 33.36

IMRC: 17.91

PSNR: 33.54

Plenoxels

Plenoxels

+ CF loss

+ CF loss

and the density field becomes sharper (marked by green circles). This is reasonable because the observation colors of the points in wrong geometry, e.g. , floaters or a thick surface, tend to be very high frequency. The SH of degree 2 fails to fit such a high-frequency color variation, and the estimated color field may produce poor rendering results, as represented by a high CF loss. During back-propagation, the CF loss can guide to a better density field.

For DVGO, the best results for all metrics on all datasets are obtained when the CF loss is added or both CF loss and distortion loss are added. These two losses complement each other's advantages, but the CF loss is more sensitive to specific geometric errors. We visualize the rendered images, depth, and residual color of several scenes for better comparison. As shown in Fig. 7, the DVGO fails to recover two details of a house as marked by blue and red rectangles. The distortion loss relieves the problems, but the windows in the red rectangle are still distorted. We try to further increase the weight factor of the distortion loss but this does not help. In contrast, our loss significantly improves the quality of the density field as shown by the depth maps and residual colors. Two more examples are shown in Fig. 8. It is noteworthy that the scene with a mic is trained with abundant views (100 views) uniformly distributed in the upper hemisphere. Under a black background, however, it severely suffers from the shape-radiance ambiguity problem. A part of the wire is confused with the background, and lots of floaters exist around the wire. The distortion loss makes no difference as it is not aware of the geometry of the scene. The CF loss successfully distinguishes the wire and background, producing a better density field. The right scene in Fig. 8 is another example. The DVGO, with or without distortion loss, mistakenly reconstructs the upper surface of the bottom rock. This mistake is avoided by our method. Overall, the CF loss effectively improves the geometry of a scene qualitatively and quantitatively.

DVGO

+ Distortion loss

+ CF loss

Rendered image

Residual color

IMRC: 17.45

PSNR: 27.96

IMRC: 17.76

PSNR: 27.78

IMRC: 18.53

PSNR: 28.64

Rendered depth

更座座

拍

DVGO

+ Distortion loss

+ CF loss

PSNR: 28.41

IMRC: 8.09

PSNR: 28.40

IMRC: 8.44

PSNR: 34.01

IMRC: 16.29

Rendered

image

Residual

color

Rendered

depth

DVGO

+ Distortion loss

+ CF loss

PSNR: 28.38

PSNR: 28.63

PSNR: 28.77

IMRC: 17.43

IMRC: 17.95

IMRC: 19.83

5.4 Computational Cost

In this subsection, we analyze the computational cost. Specifically, in each training step, a batch of B rays are used. Then, only the colors of the voxels intersected with these rays need to be estimated. Suppose that each ray involves the order of N voxels, there are M = BN voxels. To estimate the color of a voxel, the density values along the ray from the voxel to K source cameras need to be calculated. Then the total number of voxels involved is of order KMN = KBN 2 . In our implementation, each CUDA block handles one of the M voxels, and each thread handles one of the K cameras. As the threads are executed in parallel, in theory, this

reduces the computation overhead from the order of KBN 2 to N . In practice, we observe that sometimes the computational overhead is affected by the batch size. This is due to the hardware limit of the maximum CUDA threads. When an excessive number of threads are used, they are not guaranteed to be executed in parallel. In Table 3, we report the computational cost of training Plenoxels by using different batch sizes for regularization. The number of source cameras K used are 44 or 58 for the DTU, 100 for NeRF synthetic, and from 17 to 54 for LLFF dataset. The DTU dataset requires longer training time because it is trained for 12 epochs, while the NeRF synthetic dataset is trained for 9 epochs. For 9 epochs, the DTU dataset costs about 24 and 32 minutes for 10 and 25 rays, respectively. Compare these with those of NeRF synthetic dataset, we can observe that the large number of K = 100 cameras does not obviously increase the computational burden.

6 Conclusion

The shape-radiance ambiguity is a significant problem of NeRFs. Even with a simple background and abundant training images, wrong geometry can be learned and poor novel-views can be rendered. In this paper, we propose a new regularization method to break the entanglement of density and color field. The key is a closed-form color estimation method, which can recover the color field of a scene given a density field and a set of posed training images. We overcome the difficulties in estimating the color fields by handling the occlusion and non-uniformly distributed views. The image quality rendered by our closed-form color field approaches the trained one. Then, we use the photometric loss derived from the estimated color field to provide an independent supervision signal for the density field. Both the PSNR and IMRC of two explicit NeRFs, including Plenoxels and DVGO, are improved. Experimental results show the capability of the CF loss to correct geometric errors compared with existing volume and ray-based losses.

PSNR: 26.47

PSNR: 23.91

Closed-form color field

Ground truth

Limitations and future work . The closed-form color field has difficulties to perfectly recover highly reflective objects. As shown in Fig. 9, the surface of the scissors is less reflective compared to the ground truth. Although increasing the SH degree can relieve this problem to some extent as shown in Appendix B.1.2, it needs to be solved more systematically. For example, the probability density function p d ( d ) of directions can be better estimated, and a rigorous proof of the bias reduction strategy needs to be further studied. Besides, a well-estimated color field has the potential to get rid of the parameterized color field in a NeRF, and then only the density field needs to be trained and stored.

References

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 5460-5469. IEEE, 2022.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial Radiance Fields. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII , volume 13692 of Lecture Notes in Computer Science , pages 333-350. Springer, 2022.

Zhiqin Chen, Thomas A. Funkhouser, Peter Hedman, and Andrea Tagliasacchi. MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , pages 16569-16578. IEEE, 2023.

Boyang Deng, Jonathan T. Barron, and Pratul P. Srinivasan. JaxNeRF: An Efficient JAX Implementation of NeRF, 2020.

Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised NeRF: Fewer Views and Faster Training for Free. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 12872-12881. IEEE, 2022.

Qihang Fang, Yafei Song, Keqiang Li, Li Shen, Huaiyu Wu, Gang Xiong, and Liefeng Bo. Evaluate Geometry of Radiance Field with Low-frequency Color Prior. arXiv preprint arXiv:2304.04351 , 2023.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance Fields Without Neural Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 5491-5500. IEEE, 2022.

Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien P. C. Valentin. FastNeRF: High-Fidelity Neural Rendering at 200FPS. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 14326-14335. IEEE, 2021.

Robin Green. Spherical Harmonic Lighting: The Gritty Details. In Archives of the game developers conference , volume 56, page 4, 2003.

Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul E. Debevec. Baking Neural Radiance Fields for Real-Time View Synthesis. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 5855-5864. IEEE, 2021.

Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. EfficientNeRF - Efficient Neural Radiance Fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 12892-12901. IEEE, 2022.

Rasmus Ramsbøl Jensen, Anders Lindbjerg Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large Scale Multi-view Stereopsis Evaluation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014 , pages 406-413. IEEE Computer Society, 2014.

Mijeong Kim, Seonguk Seo, and Bohyung Han. InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 12902-12911. IEEE, 2022.

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural Sparse Voxel Fields. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Stephen Lombardi, Tomas Simon, Jason M. Saragih, Gabriel Schwartz, Andreas M. Lehrmann, and Yaser Sheikh. Neural Volumes: Learning Dynamic Renderable Volumes from Images. ACM Transactions on Graphics , 38(4):65:1-65:14, 2019.

Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 7210-7219. Computer Vision Foundation / IEEE, 2021.

Nelson L. Max. Optical Models for Direct Volume Rendering. IEEE Transactions on Visualization and Computer Graphics , 1(2):99-108, 1995.

Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan, and Jonathan T. Barron. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 16169-16178. IEEE, 2022.

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Transactions on Graphics , 38(4):29:1-29:14, 2019.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I , volume 12346 of Lecture Notes in Computer Science , pages 405-421. Springer, 2020.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics , 41(4):102:1102:15, 2022.

Malte Prinzler, Otmar Hilliges, and Justus Thies. DINER: Depth-aware Image-based NEural Radiance fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , pages 12449-12459. IEEE, 2023.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 14315-14325. IEEE, 2021.

Gernot Riegler and Vladlen Koltun. Stable View Synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 12216-12225. Computer Vision Foundation / IEEE, 2021.

Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 12882-12891. IEEE, 2022.

Leonid I. Rudin and Stanley J. Osher. Total Variation Based Image Restoration with Free Local Constraints. In Proceedings 1994 International Conference on Image Processing, Austin, Texas, USA, November 13-16, 1994 , pages 31-35. IEEE Computer Society, 1994.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 5449-5459. IEEE, 2022.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Improved Direct Voxel Grid Optimization for Radiance Fields Reconstruction. arXiv preprint arXiv:2206.05085 , 2022.

Mikaela Angelina Uy, Ricardo Martin-Brualla, Leonidas J. Guibas, and Ke Li. SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , pages 16518-16527. IEEE, 2023.

Dor Verbin, Peter Hedman, Ben Mildenhall, Todd E. Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 5481-5490. IEEE, 2022.

Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed Neural Voxels for Fast Multi-view Video Synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 19706-19716, October 2023.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for Real-time Rendering of Neural Radiance Fields. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 5732-5741. IEEE, 2021.

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. NeRF++: Analyzing and Improving Neural Radiance Fields. arXiv preprint arXiv:2010.07492 , 2020.

Appendix

A More Explanations for the Residual Color Estimation Scheme

We first shed light on why the residual color estimation scheme helps reduce the bias caused by nonuniform sampling. Then we conduct numerical experiments to support our claim. For simplicity, the explanation is based on using Fourier series to approximate one-dimensional signals. The basic idea is the same and can be easily extended to the case that uses spherical harmonics (SH) to approximate the signals defined on a spherical surface.

Suppose that the target signal f ( x ) defined on [ -π, π ] is 'good' enough, or satisfies the Dirichlet Conditions, so it could be expanded into Fourier series, i.e. ,

Given a set of samples { ( x t , f ( x t )) | t = 1 , ..., T } , we aim to estimate the Fourier coefficients up to a typical degree k max . The calculation of the coefficient a k is well-known as

Using the Monte Carlo method, the above integral can be approximated by T samples x 1 , ..., x T as

Because of the orthogonal completeness of the trigonometric function set, if uniform sampling is performed and the number of samples is large enough, all other terms in Eq. (15) will approach 0 except for a k cos( kx t ) cos( kx t ) . Moreover, according to the Monte Carlo method, we have

And then,

In the nonuniform sampling case, however, non-negligible estimation biases will be generated. As an example, we visualize the bias generated by the direct current component 1 T ∑ T t =1 A 0 cos( x t ) in Fig, 10. In the uniform sampling case, 1 T ∑ T t =1 A 0 cos( x t ) is better approaching 0 , as the positive (red) and negative (blue) function values are well counteracted. In contrast, the nonuniform sampling case induces a larger bias as controlled by the magnitude of A 0 . We show such a bias in a numerical experiment later.

To reduce the bias caused by ∑ T t =1 A 0 cos( kx t ) , we find it effective to subtract this component in the estimation, i.e. ,

-𝐴0

𝐴0

0

0.5𝐴0

-0.5𝐴0

Uniform sampling

Nonuniform sampling

1

𝑇 ෍

𝑡=1

𝑇

𝐴0 cos 𝑥𝑡

= 0.09𝐴0

1

𝑇 ෍

𝑡=1

𝑇

𝐴0 cos 𝑥𝑡

= 0.39𝐴0

<

2

2

This time the Monte Carlo estimator of a k becomes

Compared with ˆ a k defined in Eq. (15), ˆ a ′ k completely eliminates the bias caused by ∑ T t =1 A 0 cos( kx t ) . Thus, the Monte Carlo estimator

as derived from 1 π ∫ π -π ( f ( x ) -A 0 ) cos( kx ) dx suffers less bias from nonuniform sampling than

as derived from 1 π ∫ π -π f ( x ) cos( kx ) dx . Because A 0 is actually unknown, we first estimate this direct current component using the Monte Carlo method as

Then, we substitute the unknown A 0 with ˆ A 0 in Eq. (20).

Note that, the bias of other components that are integrated to 0 could be analysed and eliminated similarly. We show in the experiments later that such a treatment effectively reduces the estimation bias.

As a summary, when estimating the coefficient a k of a frequency component, we should subtract all other frequency components that are integrated to 0 to reduce the estimation bias as much as possible. Notably, the coefficients for frequency components k +1 , k +2 , ..., k max are unknown if we perform the estimation in turn from 1 to k . One feasible solution is to perform the estimation iteratively. Concretely, in the first round, we only subtract the frequency components 1 to k -1 when estimating a k . Then in the second round, we can subtract all other components. The iteration is ended until the difference of the estimated coefficients in the adjacent round becomes small enough. In the main paper, only one round of estimation is performed considering the computational overheads. Notice that because the SH functions also has orthogonal completeness, the above scheme, i.e. , estimating the coefficients in turn and eliminating the influences of other frequency components, also makes a difference for estimating SH coefficients as we have done in the main paper .

We perform numerical experiments about Fourier series to support our claim. We compare three estimators, including the original Monte Carlo estimator, Monte Carlo estimator using our bias

reducing strategy, and the estimator based on the least square method. Specifically, the least square solution is

where † denotes pseudo-inverse. We make sure that the number of samples T is larger than the number of coefficients to be estimated.

For the target signals, we randomly choose from trigonometric and polynomial functions, and determine their coefficients randomly. As a result, three random target functions are chosen as follows,

To perform nonuniform sampling, we sample x t from a normal distribution N (0 , σ 2 ) . To compare the three estimators quantitatively, we uniformly sample 1000 points from [ -2 σ, 2 σ ] . Then, we calculate the root mean squared error (RMSE) of this 1000 points. Because of the randomness of the estimation process, we repeat the estimation for 100 , 000 times and report the mean RMSE (MRMSE) as the final metric, i.e. ,

where x ij is the j th sample in the i th estimation process. f ( · ) is the ground-truth signal and ˆ f i ( · ) is the estimated signal in the i th estimation process. The lower the MRMSE, the better.

We set k max = 3 , σ = 10 , and the number of samples for estimation T = 10 , 20 , ..., 100 . The MRMSE results for f 1 ( x ) , f 2 ( x ) , and f 3 ( x ) are shown in Fig. 11. It is clear that with our bias reducing strategy, the estimation is consistently better than original Monte Carlo estimation. Furthermore, in the case of sparse sampling, the least square method suffers a sharp performance degradation. When only 10 samples are available, it is even worse than the original Monte Carlo estimation. When more samples are given, our method and the least square method become comparable.

MRMSE

Least square

𝑓 1 (𝑥)

𝑓 2(𝑥)

𝑓 3 (𝑥)

+

30

60

10.0

28

7.5

40

5.0

20

2.00

10

6

1.75

9

5

1.50

1.25

8

4

1.00

7

20

40

60

80

100

20

40

60

80

100

20

40

60

80

100

Number of samples for coefficient estimation ( 𝑇 )

In addition, we add different direct current values to f 1 ( x ) to increase its A 0 , and fix T = 100 . The MRMSE results are listed in Table 4. It shows that the estimation bias of Monte Carlo estimation increases with a larger A 0 , while our method and the least square method are invariant about such additions.

Finally, the computational cost of our method, least square method, and original Monte Carlo method for one estimation with 100 samples are 1 . 196 × 10 -4 s, 1 . 052 × 10 -4 s, and 0 . 978 × 10 -4 s respectively. The experiments are performed on an Intel i7-9700 CPU. Overall, we can conclude that our method effectively reduces estimation biases with acceptable computational overhead.

B More Experimental Results

B.1 More Closed-Form Color Estimation Results

B.1.1 Probability Density Functions for Directions

Acommon choice for modeling the distribution of 3D directions is to use the von Mises-Fisher (vMF) distribution defined as follows,

where µ is the normalized mean direction, and c is the concentration parameter. Because the vMF distribution only deals with the unimodal case, we resort to the mixture of von Mises-Fisher distribution defined as

where d 1 , ..., d K are known viewing directions as the modes of the distribution. It is noteworthy that when c approaches zero, the probability density function becomes a uniform distribution. Thus, the uniform distribution used in the main paper is a special case of the mixture of vMF distribution. The concentration parameter c is a hyper-parameter. We vary it, and use the probability density function to estimate the color field of DTU scenes given the density field from trained DVGOs. The PSNR results of estimated color fields are reported in Table 5. It shows that a large concentration parameter decreases PSNR. Although c = 0 . 01 is slightly better by 0.01 PSNR, we apply a uniform distribution ( c = 0 ) in the main paper for its simplified and more efficient calculation process.

B.1.2 Different SH Degrees

In Table 6, we report the PSNR of estimated color fields using different SH degrees. It shows a clear trend that the PSNR increases with the increase of the SH degree. The computational cost is not obviously affected by the SH degree, but the higher SH degree requires much memory.

Table 6: PSNR of estimated color fields using different SH degrees based on DVGO's density fields.

B.2 More Regularization Results

B.2.1 Visualization and Comparison of Ray-based Losses

We train the DVGO [15, 27] on a typical scene (DTU scan 40). After training, we calculate common ray-based losses, including the photometric loss, per-point RGB loss [27], background entropy loss

The existing ray-based losses are not aware about the geometry.

11.0%

PSNR: 31.48

4.7%

11.1%

6.5%

12.4%

PSNR: 26.42

28.0%

[15, 27], ray-entropy loss [13], and distortion loss [1], of a training view based on the trained model. For ray-based losses, each pixel has a loss value, so we can visualize the losses as images as shown in Fig. 12. We marked a region with a red rectangle, where the top surface of the bottom rock is falsely reconstructed. On the left-top corners of the loss images, we list the percentages of the losses in the red rectangle over the whole image. These percentages reflect to what extent the corresponding losses focus on the geometric error. The photometric loss of the training view has a very low percentage of 4.7%. This demonstrates that minimizing the photometric loss cannot assure a correct geometry, which is caused by the shape-radiance ambiguity problem. For other ray-based losses, they are also not sensitive enough to the false geometry, with the largest percentage of 12.4%. In contrast, the CF loss has a significantly larger percentage of 28%. It focuses more on geometric errors than common ray-based losses. The errors can be reflected by the image rendered by the closed-form color field, as shown by the right-top image of Fig. 12. The red rectangle has a large closed-form photometric loss, and, more importantly, this loss can be used to improve the geometry of the scene.

B.2.2 Effect of the Weight Factor λ and Number of Rays |R cf |

We vary the two hyper-parameters, including the weight factor and number of rays, on Plenoxels [7], and report the average PSNR and IMRC of the DTU dataset in Table 7. The PSNR and IMRC have an increasing trend when we increase the weight factor. Moreover, we showcase an example scene in Figs. 13-14. By increasing the weight factor or the number of rays, the problem of thick surface has been relieved. When the weight factor is too large ( λ = 20 ), however, the PSNR decreases. This is because the CF loss dominates the training. While it only provides the supervision signals to the density field, the color field cannot be well optimized. Therefore, the weight factor cannot be too large.

For the number of rays, increasing it helps improve the geometry of the scene. However, more rays mean higher computational cost. To be more specific, 17 minutes are needed on average to train the DTU scenes without CF loss. When we add the CF loss and use 10 rays in every iteration, 32 minutes are needed on average. The time cost further increases to 43 minutes if we use 25 rays. To ensure the training efficiency, we do not increase the number of rays further.

PSNR: 27.72

PSNR: 27.79

PSNR: 27.81

PSNR: 27.79

IMRC: 14.96

IMRC: 15.46

IMRC: 16.10

IMRC: 16.47

IMRC: 16.73

PSNR: 27.73

𝜆 = 0

𝜆 = 1

𝜆 = 5

𝜆 = 10

𝜆 = 2

0

Rendered

image

Residual

color

Rendered

depth

PSNR: 27.72

IMRC: 14.96

|ℛ𝑐𝑓 | = 0, 𝜆 = 0

Rendered

image

Residual

color

Rendered

depth

|ℛ𝑐𝑓 | = 10, 𝜆 = 10

|ℛ𝑐𝑓 | = 25, 𝜆 = 10

PSNR: 27.79

IMRC: 16.47

PSNR: 27.74

IMRC: 16.20

|ℛ𝑐𝑓 | = 10, 𝜆 = 1

|ℛ𝑐𝑓 | = 25, 𝜆 = 1

PSNR: 27.79

IMRC: 15.46

PSNR: 27.79

IMRC: 15.37

国

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper use closed-form spherical harmonics (SH) color-field to regularize the training of NeRF-based models. The SH coefficient of each 3D point has a closed-form solution given a set of observed viewing angles. This work further takes the transmittance from the density field to weigh each observed angle. A residual estimation is proposed to reduce the bias due to non-uniform view angles sampling from the training views. The close-form color-field is only used to regularize the optimization of density field. Experiments show that the proposed regularization can improve quality and alleviate shape-raidance ambiguity.\n\nSTRENGTHS:\nIntroducing the closed-form SH coefficent solver into NeRF's optimization is an interesting aspect to improve quality of NeRF-based models. I'm interested to see if it is possible to have a closed-form optimal-PSNR color-field in the future given a fixed geometry.\n\nWEAKNESSES:\n1. The manuscript incorrectly use the term \"volumetric-based\" to differentiate voxel-based methods from MLP-based methods (L76). Both of them are volumetric representations mapping spatial coordinates to modality of interest. Please fix it using \"implicit\" and \"explicit\" representations instead.\n\n2. Some baseline results are much worse than their official report. The reported PSNR of Plenoxel on NeRF synthetic dataset is 29.83 which is much lower than their official reported 31.71 in Plenoxel's Table2. The baseline DVGO on Mic scene in Fig8 is much worse than their official results where DVGO doesn't produce the floater artifact around the mic wire. I guess the reason is the background color (from solid white to solid black) which make the shape-radiance ambiguity more severe on some scenes.\nIt is neccessary to have a discussion about the discrepancy of the implementations and the results comparing to the original baselines.\n\n3. The proposed closed-form color-field solution is point-wise without considering the colors from the other points on a ray. As a result, the solved color-field may not be the optimal regarding the photometric loss. This may be one of the reason why Plenoxel (which uses SH as well) have a worse PSNR when using the proposed closed-form solution.\n\n4. Lack of theoratical backup about the bias reduction using residual. The eliminated components (supp's Eq.6) itself is estimated (supp's Eq.10) and is biased due to the non-uniform sampling. It is not clear to me how the overall bias is reduced by using the residual.\n\n5. The ablation study only use a single scene (Fig.4). More examples can make it more convincing.\n\n6. The quantitative improvement is incremental while it needs much more training time (1 hour vs. 10 minutes) and specialized CUDA implementation.\n\nQUESTIONS:\n1. I still don't quite convinced why Plenoxel's color field get much worse PSNR when using the proposed closed-form solver comparing to the SGD solver (Fig5,L235-240). As the original Plenoxel use SH as well, I expect the closed-form solver to have a even better or similar MSE (PSNR) under the same fixed density field. Are there any other factors like the degree of SH or the grid resolution affecting the results? More discussions are necessary.\n\n2. Missing experiments for the number of SH degree. The only discussion is Fig9 showing failure cases on high-frequency view-dependent effect. What is the quality improvement and computation cost of using more or less SH degree?\n\n3. This work assumes the view sampling is uniform so the Monte Carlo integrator (Eq.5) has large error when the training viewing angles is not uniform (Fig.3). However, we actually know the camera poses. Is it possible to derive a pdf over viewing angles using the camera poses so that we can have a correct Monte Carlo integrator like: $$\\frac{1}{K} \\sum_{k=1}^K \\frac{f(x_k)}{\\mathrm{pdf}(x_k)} ~.$$\n\nLIMITATIONS:\nThe weakness doesn't discussed the discrepency between the point-wise assumption and the volume rendering (see weakness 3).\n\nIf the authors decide to give up theoratical proof for the proposed bias reduction strategy, it should also be listed as one of the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile NeRFs achieve photorealistic results, they suffer from the shape and radiance ambiguity that is inherent in joint reconstruction. To alleviate this issue, the authors aim to separate the estimations using a closed-form radiance estimation. They leverage Spherical Harmonics to represent the radiance given a density field. With explicit occlusion handling based on the density field, all cameras with visibility for the point can be detected. The colors are then used to fit the SH coefficients for the specific surface point. The authors propose to use this closed-form color to regularize the NeRF training by enforcing similar behavior between the NeRF output and the closed-form color besides the regular photometric loss. \n\nThe effectiveness of this formulation is shown in various experiments and is agnostic to the underlying NeRF method used.\n\nSTRENGTHS:\n- One of the most significant advantages of this method is that the regularization is universally applicable to any NeRF method. \n- The method introduces a nice way to add a soft global view-dependent color regularization. \n- Besides issues with highly reflective surfaces, it improves considerably.\n\nWEAKNESSES:\n- Currently, I have the feeling the paper would benefit from an overview figure. This can also be beneficial as a reference in the experiments, where some evaluations only rely on the closed-form color, but in others, it's used as a regularization. Here, one can create a direct link to the overview figure and the symbols for each color output. A simple overview of how the method slots in an existing NeRF framework would be a great addition and can be easily incorporated next to Fig. 1.\n- The authors compared against other ray distribution priors but did not compare with model priors such as RegNeRFs depth/normal input from omnidata.  \n- Similarly, RefNeRF also aims to improve surface reconstruction. How does this method fare against the proposed one?\n- The authors should briefly discuss Riegler et al. - Stable View Synthesis, as it also handles aggregation of features from multiple views. The approach is vastly different, but the rough conceptual idea remains.\nThe authors mentioned that the closed-form color field has issues approximating highly reflective objects, but the limitations are also quite severe. The performance of anything which drastically derivates from Lambertian reflection will suffer. Biasing everything towards Lambertian also solves the shape/radiance ambiguity. So I would like to see if experiments with only lower frequency view encodings or a RefNeRF with an l2 penalty on the estimated roughness are used.\n\nMinor:\n- In 81/82: NeRF’s representation _of_ a scene\n- In 82: remove ‘makes it’: The NeRF’s representation of a scene inherently suffers from shape-radiance ambiguity\n- In 83: error _in_ geometry.\n- In 86: needs to be solved\n- In 174, the authors used minus as a verb. The correct verb is: subtract\n\nQUESTIONS:\n- What is the additional overhead of the closed-form color calculation?\n- Would the decrease in performance on highly reflective surfaces be mitigated with more SH bands, or would the method become instable during training?\n\nLIMITATIONS:\nSee the last point in the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile nerfs have shown impressive properties in the ability to reconstruct the geometry of three dimensional scenes, there are still some cases where the performances are not as good as expected. This paper focuses on one of these problems, the untangling of the geometry from the color. For that, the authors propose to learn a second color model, using a close form formula based on a spherical harmonics model. On top of the classic ray tracing of nerf, this method requires to trace additional rays that goes from the query point to the cameras. The authors also show the existence of a bias depending on the distribution of view angle and propose a method to correct it.\n\nSTRENGTHS:\nThe paper focus on an important problem. Estimating accurately the geometry in complex scene, especially with high specularity, is still a very relevant today.\n\nDespite the heavy computational load, the authors were able to successfully train different models. They also provide results on three different datasets, with two of them being real images.\n\nWEAKNESSES:\nThe premise of the method is that density and colors are estimated independently in a nerf model. This is actually false. In particular, in the vanilla nerf model the prediction of the color directly depends on the density. The color MLP takes as input not the coordinates of the point but the features corresponding to the density predicting network. This means that nerf should actually be modeled by (c) in Figure 1 and not (a) as claimed in the paper.\n\nMany variables are not introduced properly making the reading more difficult that it should.\n\nThe method is highly inefficient. For a given ray, for a given sampled point, the method requires to integrate the density along the ray toward each source camera. This means that if the model is trained using $N$ cameras and $M$ points are sampled each time to approximate the integrals, the proposed method requires to estimate the density of $NM^2$ points instead of the usual $M$ points.\n\nThe parameters of the method vary between datasets. No explanation or intuition is provided for that.\n\nRegarding the experiments, the results are quite incremental with an increase of the order of 0.1dB compared to two previous methods. It's also disappointing that the analysis was not performed with more models. We can especially mention tensoRF that also has a model based on SH. It would have also been interesting to compare to methods like regnerf that adds additional regularizers to nerf to be able reconstruct better the geometry even in difficult conditions. Moreover, while the paper focuses on improving the geometry of the trained model, there is no quantitative experiment showing that the proposed method has indeed an impact on the geometry thus learned. Only a few qualitative results are presented with a color map for the depth that is very difficult to interpret. One can also regret the lack of computational analysis. No study are done on the unbiaising process proposed in the paper.\n\nQUESTIONS:\nWhat strategy of point sampling is performed for estimating the color and transmittance of a given point (i.e when the ray from this point to the camera is drawn)?\n\nLIMITATIONS:\nI don't foresee any potential negative societal impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel regularization technique for optimizing radiance fields to achieve more accurate geometry alongside realistic novel views. The regularizer is based on a closed-form estimation of spherical harmonic coefficients for view-dependent color, as a function of the training pixel colors and the predicted density field. Using this closed-form procedure to estimate color from density enables regularizing density directly through photometric loss. This regularizer is therefore adaptive to each scene, unlike more general, standard scene-agnostic regularizers like sparsity and total variation. The paper shows empirical improvement in reconstructed geometry via improved rendered depth maps, for two voxel-based scene representations (DVGO and Plenoxels).\n\nSTRENGTHS:\nI really like the observation the paper makes that it is possible to predict color in closed form as a function of density field and training ray colors. It makes a lot of sense that using this closed-form solution for color and then optimizing density directly through photometric loss, should yield more accurate reconstructed density fields. And indeed the results do show improved depth renderings.\n\nWEAKNESSES:\npresentation weaknesses\n- Figure 2 should include o_k and P_k, and ideally also F_c. In its current form there is a bit of a struggle to match variables in the text to components in the figure.\n- I don’t see much value added by Figure 3. It is used to illustrate the idea that nonuniform or systematically biased sampling can result in a systematic error in the estimated integral. In my view, this idea is clear from textual description, and the space currently dedicated to the figure could be better spent on detail (either textual or ideally graphical) about how the proposed bias correction works.\n- As an added point of confusion around Figure 3, why include a term that is cos(0x) (in both the figure and the accompanying text)? It appears that this is literally referring to cos(0)=1, which has no effect in the example shown.\n- Equation 14 (the definition of the IMRC metric) could benefit from another sentence of explanation, describing the intuition that this metric is an approximation of PSNR in 3D.\n- Figures 4 and 5 would benefit from more substantial captions that explain the experiment being presented and how the results should be interpreted. Currently this information is only available in the text, but it’s best if figures are also self-contained.\n- It’s not clear from the paper whether the results (or which subset of the results) are fine-tuned using pre-trained models vs trained from scratch. I’m not sure if the argument is that fine-tuning with the closed-form loss improves existing pre-trained geometry, or that training from scratch with the closed-form loss as a regularizer produces better geometry, or both.\n- There are scattered typos and minor grammatical issues; please copy-edit the final paper.\nevaluation weaknesses\n- The evaluation uses a black background, whereas in the baseline papers a white background was used. Is there a reason for changing the background color?\n- PSNR values reported for the two baselines (Plenoxels and DVGO) are substantially lower than the values reported in the original papers (29.83 vs 31.71 for Plenoxels, and 31.58 vs 31.95 and 32.8 for DVGO). As far as I’m aware the only difference is the use of the black background; I’m concerned that either this or some other implementation difference makes the baselines used unnecessarily weak.\n- I understand the IMRC metric and see why it’s useful for datasets where ground truth geometry is unavailable, but in my view an even stronger evaluation would be to compare PSNR on depth maps for a dataset where ground truth geometry is available (e.g. this could be rendered for the NeRF-synthetic dataset).\n- Figure 9 shows that there is a gap in quality between renderings produced by the trained color field vs the closed-form color field—can the authors comment on the origin of this gap, and how it might be closed?\n\nQUESTIONS:\nMy biggest question about this paper is why the proposed method is used as a regularization scheme on top of the usual least squares objective, rather than optimizing a density-only model and directly using the colors from the closed-form predictions. I wouldn’t be surprised if doing so yields even better geometry, and reduces model size as an added bonus. Or if this simpler strategy works worse than what is proposed, do the authors have an explanation why (perhaps related to the limitation illustrated in Figure 9)?\n\nI would also appreciate responses to my comments in the weaknesses section, primarily regarding evaluation weaknesses.\n\nLIMITATIONS:\nYes, limitations are discussed to some extent, but the paper would benefit from more discussion of the origin (and potential remedies, if they exist) of the limited quality of the estimated color field.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}