SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The occlusion handling and residual estimation play significant roles. We conduct an ablation study here. The results based on Plenoxels' density fields are reported in Table 1. In Fig. 4, we additionally showcase an example based on a DVGO's density field. Specifically, when we directly estimate SH coefficients according to Eq. (7) without using the residual estimation scheme, the rendered images look bad as shown by the first two sub-figures. This is because estimating the SH coefficient of one frequency band is affected by other frequency components if we cannot uniformly sample directions from the 2D sphere. Since the images of each DTU scene are approximately taken from a half sphere, the views are non-uniformly distributed. As a result, the estimation of SH coefficients suffers from biases. On the other hand, when we estimate
without occlusion handling, i.e., without using transmittance to weight an observation, the estimation result is blurred as shown by the third sub-figure. This is because in the estimation,  $\mathcal{F}_{\mathbf{v},\mathbf{d}_k}^c$  of different  $\mathbf{d}_k$  is not guaranteed to be the radiance emitted from the same point  $\mathbf{v}$ , but possibly that of a point
Table 1: Ablation study on DTU scenes based on Plenoxels' density.
Occlusion handling	Residual estimation	PSNR
14.61
✓	13.57
✓	25.99
✓	✓	26.49
w/o both
w/o residual estimation
w/o occlusion handling
w/both
DVGO
Figure 4: Ablation study on DTU scan 122 based on DVGO's density.
Figure 5: Color field estimation results of DTU scan 63.
nearer to a camera. When both occlusion handling and residual estimation are applied, as shown in the fourth sub-figure, the rendering result is visually acceptable, and quantitatively, has a much higher PSNR that approaches the trained one. To summarize, both the table and figure show that the residual estimation improves the PSNR considerably. With only occlusion handling, the PSNR decreases, but when it is combined with the residual color estimation, the PSNR is further improved.
The estimation results are not only affected by the algorithm, but also by the quality of the density field. Specifically, the estimation is based on an assumption that, if a point is not occluded, its radiance along a direction is directly determined by the corresponding observation color in the image. This assumption is true for the ideal case, where the density is sharp enough and the color reduces to the surface light field [33]. By using this assumption, the closed-form color field becomes sensitive to false geometry. If the geometry or the density field is not correct or sharp enough, which breaks this assumption, the color estimation results will be bad. In Fig. 5, we show the estimation results based on the density field of DVGO and Plenoxels, respectively. We also visualize the depth map of the scene and render the residual color. The Plenoxels suffers a sharper PSNR drop from the trained image (34.06) to estimated one (29.09). This is because it has an inferior density field as qualitatively shown by the depth maps. The errors in the density field affect the estimation process and result in poor rendering quality. In contrast, a better density field given by the DVGO permits a higher PSNR (33.08) of the estimated color field. Note that the closed-form color field is more sensitive to the quality of density field than the trained one, as it produces a larger PSNR difference  $(3.99 > 1.35)$  between two methods. By using the closed-form photometric loss, the rendering errors caused by an inferior density field can be back-propagated to improve the density field.
5.3 Results of Regularization and Comparison
In this subsection, we analyze the regularization results. Specifically, we apply our regularization to Plenoxels and DVGO and train from scratch. Based on DVGO, we additionally add a fine-tuned distortion loss for comparison. Table 2 reports the PSNR, IMRC, and PSNR of depth.
For Plenoxels, the CF loss improves all metrics on all datasets. It effectively improves the geometry of a scene. As shown in Fig. 6, the floaters in the free space (marked by red circles) are penalized,
Table 2: Comparison of PNSR  $\uparrow$  /IMRC  $\uparrow$  /(PSNR of depth↑) on the three datasets (Best).
Method	DTU	NeRF synthetic	LLFF
Plenoxels [7]	31.86 / 15.88	29.83 / 16.55 / 22.70	26.30 / 19.15
Plenoxels [7] + CF loss	32.08 / 16.66	29.84 / 16.59 / 23.08	26.40 / 20.02
DVGO [27, 28]	32.15 / 18.19	31.58 / 17.25 / 25.46	26.23 / 18.87
DVGO [27, 28] + Distortion loss [1]	32.20 / 18.47	31.50 / 17.93 / 25.68	26.33 / 20.76
DVGO [27, 28] + CF loss	32.23 / 18.51	32.24 / 17.61 / 26.36	26.26 / 19.66
DVGO [27, 28] + Distortion loss [1] + CF loss	32.26 / 18.80	32.23 / 18.90 / 26.68	26.34 / 20.92
Figure 6: Our loss improves the geometry of a scene by making the density field sharper and removing the floaters in the free space. The two scenes are DTU scan 37 and 105 from left to right.
and the density field becomes sharper (marked by green circles). This is reasonable because the observation colors of the points in wrong geometry, e.g., floaters or a thick surface, tend to be very high frequency. The SH of degree 2 fails to fit such a high-frequency color variation, and the estimated color field may produce poor rendering results, as represented by a high CF loss. During back-propagation, the CF loss can guide to a better density field.
For DVGO, the best results for all metrics on all datasets are obtained when the CF loss is added or both CF loss and distortion loss are added. These two losses complement each other's advantages, but the CF loss is more sensitive to specific geometric errors. We visualize the rendered images, depth, and residual color of several scenes for better comparison. As shown in Fig. 7, the DVGO fails to recover two details of a house as marked by blue and red rectangles. The distortion loss relieves the problems, but the windows in the red rectangle are still distorted. We try to further increase the weight factor of the distortion loss but this does not help. In contrast, our loss significantly improves the quality of the density field as shown by the depth maps and residual colors. Two more examples are shown in Fig. 8. It is noteworthy that the scene with a mic is trained with abundant views (100 views) uniformly distributed in the upper hemisphere. Under a black background, however, it severely suffers from the shape-radiance ambiguity problem. A part of the wire is confused with the background, and lots of floaters exist around the wire. The distortion loss makes no difference as it is not aware of the geometry of the scene. The CF loss successfully distinguishes the wire and background, producing a better density field. The right scene in Fig. 8 is another example. The DVGO, with or without distortion loss, mistakenly reconstructs the upper surface of the bottom rock. This mistake is avoided by our method. Overall, the CF loss effectively improves the geometry of a scene qualitatively and quantitatively.
Figure 7: Comparison of distortion loss and CF loss on DTU scan 24.
Figure 8: Comparison of distortion loss and CF loss on the mic from the NeRF synthetic dataset and DTU scan 40.
5.4 Computational Cost
In this subsection, we analyze the computational cost. Specifically, in each training step, a batch of  $B$  rays are used. Then, only the colors of the voxels intersected with these rays need to be estimated. Suppose that each ray involves the order of  $N$  voxels, there are  $M = BN$  voxels. To estimate the color of a voxel, the density values along the ray from the voxel to  $K$  source cameras need to be calculated. Then the total number of voxels involved is of order  $KMN = KBN^2$ . In our implementation, each CUDA block handles one of the  $M$  voxels, and each thread handles one of the  $K$  cameras. As the threads are executed in parallel, in theory, this
Table 3: Training time (minutes) with different batch sizes  $(B)$
B	DTU	NeRF synthetic	LLFF
0	17.7	16.0	24.2
10	32.7	20.5	29.4
25	43.0	25.5	34.6
reduces the computation overhead from the order of  $KBN^2$  to  $N$ . In practice, we observe that sometimes the computational overhead is affected by the batch size. This is due to the hardware limit of the maximum CUDA threads. When an excessive number of threads are used, they are not guaranteed to be executed in parallel. In Table 3, we report the computational cost of training Plenoxels by using different batch sizes for regularization. The number of source cameras  $K$  used are 44 or 58 for the DTU, 100 for NeRF synthetic, and from 17 to 54 for LLFF dataset. The DTU dataset requires longer training time because it is trained for 12 epochs, while the NeRF synthetic dataset is trained for 9 epochs. For 9 epochs, the DTU dataset costs about 24 and 32 minutes for 10 and 25 rays, respectively. Compare these with those of NeRF synthetic dataset, we can observe that the large number of  $K = 100$  cameras does not obviously increase the computational burden.
The shape-radiance ambiguity is a significant problem of NeRFs. Even with a simple background and abundant training images, wrong geometry can be learned and poor novel-views can be rendered. In this paper, we propose a new regularization method to break the entanglement of density and color field. The key is a closed-form color estimation method, which can recover the color field of a scene given a density field and a set of posed training images. We overcome the difficulties in estimating the color fields by handling the occlusion and non-uniformly distributed views. The image quality rendered by our closed-form color field approaches the trained one. Then, we use the photometric loss derived from the estimated color field to provide an independent supervision signal for the density field. Both the PSNR and IMRC of two explicit NeRFs, including Plenoxels and DVGO, are improved. Experimental results show the capability of the CF loss to correct geometric errors compared with existing volume and ray-based losses.
Trained color field
Figure 9: Color estimation results of a highly reflective object given the trained DVGO density.
Closed-form color field
Limitations and future work. The closed-form color field has difficulties to perfectly recover highly reflective objects. As shown in Fig. 9, the surface of the scissors is less reflective compared to the ground truth. Although increasing the SH degree can relieve this problem to some extent as shown in Appendix B.1.2, it needs to be solved more systematically. For example, the probability density function  $p_{\mathrm{d}}(\mathbf{d})$  of directions can be better estimated, and a rigorous proof of the bias reduction strategy needs to be further studied. Besides, a well-estimated color field has the potential to get rid of the parameterized color field in a NeRF, and then only the density field needs to be trained and stored.
This work was supported in part by the National Key Research and Development Program of China under Grant 2021YFB3301504, in part by the National Natural Science Foundation of China under Grant U19B2029, Grant U1909204, and Grant 92267103, in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2021B1515140034, and in part by the CAS STS Dongguan Joint Project under Grant 20211600200022.
[1] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5460-5469. IEEE, 2022.
[30] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd E. Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5481-5490. IEEE, 2022.
[31] Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed Neural Voxels for Fast Multi-view Video Synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19706-19716, October 2023.
[32] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for Real-time Rendering of Neural Radiance Fields. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 5732-5741. IEEE, 2021.
[33] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. NeRF++: Analyzing and Improving Neural Radiance Fields. arXiv preprint arXiv:2010.07492, 2020.
A More Explanations for the Residual Color Estimation Scheme
We first shed light on why the residual color estimation scheme helps reduce the bias caused by nonuniform sampling. Then we conduct numerical experiments to support our claim. For simplicity, the explanation is based on using Fourier series to approximate one-dimensional signals. The basic idea is the same and can be easily extended to the case that uses spherical harmonics (SH) to approximate the signals defined on a spherical surface.
Suppose that the target signal  $f(x)$  defined on  $[-\pi, \pi]$  is "good" enough, or satisfies the Dirichlet Conditions, so it could be expanded into Fourier series, i.e.,
$$
f (x) = A _ {0} + \sum_ {n = 1} ^ {\infty} \left[ a _ {n} \cos (n x) + b _ {n} \sin (n x) \right]. \tag {13}
$$
Given a set of samples  $\{(x_{t},f(x_{t}))|t = 1,\dots,T\}$ , we aim to estimate the Fourier coefficients up to a typical degree  $k_{max}$ . The calculation of the coefficient  $a_{k}$  is well-known as
$$
\begin{array}{l} a _ {k} = \frac {1}{\pi} \int_ {- \pi} ^ {\pi} f (x) \cos (k x) d x \\ = \frac {1}{\pi} \int_ {- \pi} ^ {\pi} \left\{A _ {0} \cos (k x) + \sum_ {n = 1} ^ {\infty} \left[ a _ {n} \cos (n x) \cos (k x) + b _ {n} \sin (n x) \cos (k x) \right] \right\} d x. \tag {14} \\ \end{array}
$$
Using the Monte Carlo method, the above integral can be approximated by  $T$  samples  $x_{1},\ldots ,x_{T}$  as
$$
\hat {a} _ {k} = \frac {1}{\pi} \frac {2 \pi}{T} \sum_ {t = 1} ^ {T} \left\{A _ {0} \cos (k x _ {t}) + \sum_ {n = 1} ^ {\infty} \left[ a _ {n} \cos (n x _ {t}) \cos (k x _ {t}) + b _ {n} \sin (n x _ {t}) \cos (k x _ {t}) \right] \right\}. \tag {15}
$$
Because of the orthogonal completeness of the trigonometric function set, if uniform sampling is performed and the number of samples is large enough, all other terms in Eq. (15) will approach 0 except for  $a_{k}\cos (kx_{t})\cos (kx_{t})$ . Moreover, according to the Monte Carlo method, we have
$$
\int_ {- \pi} ^ {\pi} \cos (k x) \cos (k x) d x \approx \frac {2 \pi}{T} \sum_ {t = 1} ^ {T} \cos \left(k x _ {t}\right) \cos \left(k x _ {t}\right). \tag {16}
$$
And then,
$$
\hat {a} _ {k} \approx \frac {1}{\pi} \frac {2 \pi}{T} \sum_ {t = 1} ^ {T} a _ {k} \cos \left(k x _ {t}\right) \cos \left(k x _ {t}\right) = a _ {k} \frac {1}{\pi} \frac {2 \pi}{T} \sum_ {t = 1} ^ {T} \cos \left(k x _ {t}\right) \cos \left(k x _ {t}\right) \approx a _ {k} \frac {1}{\pi} \pi = a _ {k}. \tag {17}
$$
In the nonuniform sampling case, however, non-negligible estimation biases will be generated. As an example, we visualize the bias generated by the direct current component  $\frac{1}{T}\sum_{t=1}^{T}A_0\cos(x_t)$  in Fig. 10. In the uniform sampling case,  $\frac{1}{T}\sum_{t=1}^{T}A_0\cos(x_t)$  is better approaching 0, as the positive (red) and negative (blue) function values are well counteracted. In contrast, the nonuniform sampling case induces a larger bias as controlled by the magnitude of  $A_0$ . We show such a bias in a numerical experiment later.
To reduce the bias caused by  $\sum_{t=1}^{T} A_0 \cos(kx_t)$ , we find it effective to subtract this component in the estimation, i.e.,
$$
\begin{array}{l} a _ {k} = \frac {1}{\pi} \int_ {- \pi} ^ {\pi} f (x) \cos (k x) d x \\ = \frac {1}{\pi} \int_ {- \pi} ^ {\pi} (f (x) - A _ {0}) \cos (k x) d x \tag {18} \\ = \frac {1}{\pi} \int_ {- \pi} ^ {\pi} \left\{\sum_ {n = 1} ^ {\infty} [ a _ {n} \cos (n x) \cos (k x) + b _ {n} \sin (n x) \cos (k x) ] \right\} d x. \\ \end{array}
$$
Bias:  $\left|\frac{1}{T}\sum_{t=1}^{T}A_0\cos(x_t)\right| = 0.09A_0 <$
$\left|\frac{1}{T}\sum_{t = 1}^{T}A_{0}\cos (x_{t})\right| = 0.39A_{0}$
Figure 10: Nonuniform sampling generates a larger estimation bias than that of uniform sampling.
This time the Monte Carlo estimator of  $a_{k}$  becomes
$$
\hat {a} _ {k} ^ {\prime} = \frac {2}{T} \sum_ {t = 1} ^ {T} \left\{\sum_ {n = 1} ^ {\infty} \left[ a _ {n} \cos \left(n x _ {t}\right) \cos \left(k x _ {t}\right) + b _ {n} \sin \left(n x _ {t}\right) \cos \left(k x _ {t}\right) \right] \right\}. \tag {19}
$$
Compared with  $\hat{a}_k$  defined in Eq. (15),  $\hat{a}_k^\prime$  completely eliminates the bias caused by  $\sum_{t=1}^{T} A_0 \cos(kx_t)$ . Thus, the Monte Carlo estimator
$$
\frac {2}{T} \sum_ {t = 1} ^ {T} \left(f \left(x _ {t}\right) - A _ {0}\right) \cos \left(k x _ {t}\right) \tag {20}
$$
as derived from  $\frac{1}{\pi}\int_{-\pi}^{\pi}(f(x) - A_0)\cos (kx)dx$  suffers less bias from nonuniform sampling than
$$
\frac {2}{T} \sum_ {t = 1} ^ {T} f \left(x _ {t}\right) \cos \left(k x _ {t}\right) \tag {21}
$$
as derived from  $\frac{1}{\pi}\int_{-\pi}^{\pi}f(x)\cos (kx)dx$ . Because  $A_0$  is actually unknown, we first estimate this direct current component using the Monte Carlo method as
$$
\hat {A} _ {0} = \frac {1}{T} \sum_ {t = 1} ^ {T} f \left(x _ {t}\right). \tag {22}
$$
Then, we substitute the unknown  $A_0$  with  $\hat{A}_0$  in Eq. (20).
Note that, the bias of other components that are integrated to 0 could be analysed and eliminated similarly. We show in the experiments later that such a treatment effectively reduces the estimation bias.
As a summary, when estimating the coefficient  $a_{k}$  of a frequency component, we should subtract all other frequency components that are integrated to 0 to reduce the estimation bias as much as possible. Notably, the coefficients for frequency components  $k + 1, k + 2, \dots, k_{max}$  are unknown if we perform the estimation in turn from 1 to  $k$ . One feasible solution is to perform the estimation iteratively. Concretely, in the first round, we only subtract the frequency components 1 to  $k - 1$  when estimating  $a_{k}$ . Then in the second round, we can subtract all other components. The iteration is ended until the difference of the estimated coefficients in the adjacent round becomes small enough. In the main paper, only one round of estimation is performed considering the computational overheads. Notice that because the SH functions also has orthogonal completeness, the above scheme, i.e., estimating the coefficients in turn and eliminating the influences of other frequency components, also makes a difference for estimating SH coefficients as we have done in the main paper.
We perform numerical experiments about Fourier series to support our claim. We compare three estimators, including the original Monte Carlo estimator, Monte Carlo estimator using our bias
reducing strategy, and the estimator based on the least square method. Specifically, the least square solution is
$$
\left( \begin{array}{c c c c c c} 1 & \cos \left(x _ {1}\right) & \sin \left(x _ {1}\right) & \dots & \cos \left(k _ {\max } x _ {1}\right) & \sin \left(k _ {\max } x _ {1}\right) \\ 1 & \cos \left(x _ {2}\right) & \sin \left(x _ {2}\right) & \dots & \cos \left(k _ {\max } x _ {2}\right) & \sin \left(k _ {\max } x _ {2}\right) \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\ 1 & \cos \left(x _ {T}\right) & \sin \left(x _ {T}\right) & \dots & \cos \left(k _ {\max } x _ {T}\right) & \sin \left(k _ {\max } x _ {T}\right) \end{array} \right) ^ {\dagger} \left( \begin{array}{c} f (x _ {1}) \\ f (x _ {2}) \\ \vdots \\ f (x _ {T}) \end{array} \right), \tag {23}
$$
where  $\dagger$  denotes pseudo-inverse. We make sure that the number of samples  $T$  is larger than the number of coefficients to be estimated.
For the target signals, we randomly choose from trigonometric and polynomial functions, and determine their coefficients randomly. As a result, three random target functions are chosen as follows,
$$
f _ {1} (x) = 2 + 0. 0 3 x ^ {2} + 2 \sin (x) + \cos (3 x),
$$
$$
f _ {2} (x) = 1 0 - 0. 0 2 x + 0. 0 1 x ^ {2} + \cos (x) - \sin (2 x), \tag {24}
$$
$$
f _ {3} (x) = 5 + 0. 0 5 x ^ {2} - 0. 0 0 1 x ^ {3} - \sin (x) + 2 \cos (2 x).
$$
To perform nonuniform sampling, we sample  $x_{t}$  from a normal distribution  $\mathcal{N}(0,\sigma^2)$ . To compare the three estimators quantitatively, we uniformly sample 1000 points from  $[-2\sigma ,2\sigma ]$ . Then, we calculate the root mean squared error (RMSE) of this 1000 points. Because of the randomness of the estimation process, we repeat the estimation for 100,000 times and report the mean RMSE (MRMSE) as the final metric, i.e.,
$$
\mathrm {M R M S E} = \frac {1}{1 0 0 0 0 0} \sum_ {i = 1} ^ {1 0 0 0 0 0} \sqrt {\frac {1}{1 0 0 0} \sum_ {j = 1} ^ {1 0 0 0} \left(f \left(x _ {i j}\right) - \hat {f} _ {i} \left(x _ {i j}\right)\right) ^ {2}}, \tag {25}
$$
where  $x_{ij}$  is the  $j^{th}$  sample in the  $i^{th}$  estimation process.  $f(\cdot)$  is the ground-truth signal and  $\hat{f}_i(\cdot)$  is the estimated signal in the  $i^{th}$  estimation process. The lower the MRMSE, the better.
We set  $k_{max} = 3$ ,  $\sigma = 10$ , and the number of samples for estimation  $T = 10,20,\ldots ,100$ . The MRMSE results for  $f_{1}(x)$ ,  $f_{2}(x)$ , and  $f_{3}(x)$  are shown in Fig. 11. It is clear that with our bias reducing strategy, the estimation is consistently better than original Monte Carlo estimation. Furthermore, in the case of sparse sampling, the least square method suffers a sharp performance degradation. When only 10 samples are available, it is even worse than the original Monte Carlo estimation. When more samples are given, our method and the least square method become comparable.
Figure 11: The MRMSE  $\downarrow$  results for  $f_{1}(x)$ ,  $f_{2}(x)$ , and  $f_{3}(x)$ .
In addition, we add different direct current values to  $f_{1}(x)$  to increase its  $A_{0}$ , and fix  $T = 100$ . The MRMSE results are listed in Table 4. It shows that the estimation bias of Monte Carlo estimation increases with a larger  $A_{0}$ , while our method and the least square method are invariant about such additions.
Finally, the computational cost of our method, least square method, and original Monte Carlo method for one estimation with 100 samples are  $1.196 \times 10^{-4}$  s,  $1.052 \times 10^{-4}$  s, and  $0.978 \times 10^{-4}$  s respectively. The experiments are performed on an Intel i7-9700 CPU. Overall, we can conclude that our method effectively reduces estimation biases with acceptable computational overhead.
Table 4: The MRMSE  $\downarrow$  results when adding different direct current values to  ${f}_{1}\left( x\right)$  .
Addition	0	1	5	10	50	100
Monte Carlo with bias reducing	3.89	3.89	3.89	3.89	3.89	3.89
Least square	3.90	3.90	3.90	3.90	3.90	3.90
Monte Carlo	4.09	4.17	4.60	5.31	13.53	25.13
B.1 More Closed-Form Color Estimation Results
B.1.1 Probability Density Functions for Directions
A common choice for modeling the distribution of 3D directions is to use the von Mises-Fisher (vMF) distribution defined as follows,
$$
v (\mathbf {d}; \mu , c) = \frac {c}{2 \pi \left(e ^ {c} - e ^ {- c}\right) e ^ {c \mu^ {\top} \mathbf {d}}}, \tag {26}
$$
where  $\mu$  is the normalized mean direction, and  $c$  is the concentration parameter. Because the vMF distribution only deals with the unimodal case, we resort to the mixture of von Mises-Fisher distribution defined as
$$
p \left(\mathbf {d}; \mathbf {d} _ {1}, \dots , \mathbf {d} _ {K}, c\right) = \frac {1}{K} \sum_ {k = 1} ^ {K} v \left(\mathbf {d} _ {k}; \mu , c\right), \tag {27}
$$