Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation
Authors: WANG Jiaxu
Ziyi Zhang
Renjing Xu
Venue: ICLR 2024 poster (2024)
Keywords: Generalizable Neural Radiance Field; Point-based Rendering;
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: The first paradigm of Generalizable radiance field based on point-based rather than image-based neural rendering

Abstract: This paper introduces a novel paradigm for the generalizable neural radiance field (NeRF). Previous generic NeRFs combine multiview stereo techniques with image-based neural rendering, yielding impressive results, while suffering from three issues. First, occlusions often result in inconsistent feature matching. Then, they deliver distortions and artifacts in geometric discontinuities and locally sharp shapes due to their individual process of sampled points and rough feature aggregation. Third, their image-based representations experience severe degradations
when source views are not near enough to the target view. To address challenges, we propose the first paradigm that constructs the generalizable neural field based on point-based rather than image-based rendering, which we call the Generalizable neural Point Field (GPF). Our approach explicitly models visibilities by geometric priors and augments them with neural features. We propose a novel nonuniform log sampling strategy to improve rendering speed and reconstruction quality. Moreover, we present a learnable kernel spatially augmented
with features for feature aggregations, mitigating distortions at places with drastically varying geometries. Besides, our representation can be easily manipulated. Experiments show that our model can deliver better geometries, view consistencies, and rendering quality than all counterparts and benchmarks on three datasets in both generalization and finetuning settings, preliminarily proving the potential of the new paradigm for generalizable NeRF

Published as a conference paper at ICLR 2024
LEARNING ROBUST GENERALIZABLE RADIANCE FIELD WITH VISIBILITY AND FEATURE AUGMENTED POINT REPRESENTATION
Jiaxu Wang', Ziyi Zhang', Renjing Xu-1
Hong Kong University of Science and Technology (Guangzhou)
iwang457e connect www.sttttee.. en, (ziyizhang zen wng.uuinnuuuutttttteeee edu. en.
ABSTRACT
This paper introduces a novel paradigm for the generalizable neural radiance field (NeRF). Previous generic NeRFs combine multiview stereo techniques with image-based neural rendering, yielding impressive results, while suffering from three issues. First, occlusions often result in inconsistent feature matching. Then, they deliver distortions and artifacts in geometric discontinuities and locally sharp shapes due to their individual process of sampled points and rough feature aggre- gation. Third, their image-basee based representations experience severe degradations when source views are not near enough to the target view. To address challenges, we propose the first paradigm that constructs the generalizable neural field based on point-based rather than image-based rendering, which we call the General- izable neural Point Field (GPF). Our approach explicitly models visibilities by geometric priors and augments them with neural features. We propose a novel nonuniform log sampling strategy improve both rendering speed and recon- struction quality. Moreover, we present a learnable kernel spatially augmented with features for feature aggregations, mitigating distortions places with drasti- cally varying geometries. Besides, our representation can be easily manipulated. Experiments show that our model can deliver better geometries, view consisten- cies, and rendering quality than all counterparts and benchmarks on three datasets in both generalization and finetuning settings, preliminarily proving the potential of the new paradigm for generalizable NeRF.
Ground Truth
IBRNet
MVSNeRF
ENeRF
Ours
Our approach produces sharper and clearer at discontinuous geometries an unobserved scenario without per-scene training and synthesizes higher quality images than baselines.
Figure I:
INTRODUCTION
Novel view synthesis has emerged as important and widely discussed topic in computer vision and graphics. In recent years, neural rendering techniques have made remarkable progress this domain, with Neural Radiance Fields (NeRF) Mildenhall al. (2021) being prime instance. How- ever, traditional NeRF- approaches are heavily constrained by a long time for optimization and the inability to generalize unseen scenarios.

---

Published as a conference paper at ICLR 2024
Recent works such Chen al. 2021 Yu et et al. (2021) Wang et al. (222cc); Lin et al. 2002 Liu
et al. (2022); Johari et al. (2022) have investigated into generalizing NeRF to unobserved environ- ments without retraining, At present, the common motivation of existing approaches is to integrate multi-view stereo (MVS) techniques into the NeRF pipeline. Their methods are called image-based neural rendering- In this paradigm, the image features extracted by networks are projected to the query 3D points which are generated by uniform sampling in volume ray marching and regressing their colors and densities. However, the paradigm has few issues, First, geometry reasoning sig- nificantly relies on stereo matching between multi-view features. However, occlusions often destroy the feature consistencies, resulting in inaccurate geometry reconstruction. On the other hand, the sampled 3D points are often processed individually, which damages the local shape connections. In this case, distortions and artifacts commonly occur in shape-varying regions. Next, they use simple methods to aggregate features, which limits the expressive abilities of networks and causes undesirable degenerations in -ito---distribution scenes. Besides, such image-based rendering rep- resentation is not capable to interact with and edit.
To address these problems, this paper proposes a novel paradigm for generalizable NeRF based on point-based instead of image- based rendering, Generalizable neural Point Field. The proposed ap- proach includes three main components, the visibility-oriented feature fetching, the robust log sam- pling strategy, and the feanure-augmented learnable kernel. Furthermore, we present three-stage finetuning scheme. Extensive experiments were conducted on the NeRF synthetic dataset Milden- hall al. (202I). the DTU dataset Jensen al. (2014), and the BlendedMVS dataset Yao et al. (2220). The results showcase that the proposed method outperforms all benchmarks with clearer textures, sharper shapes and edges, and better consistency between views. Additionally, we show examples of point completion and refinement results to illustrate its validity in improving recon- structed geometries. Moreover, we show the potential of the method in interactive manipulation.
The main contributions of this paper are as follows:
We first propose a novel paradigm GPF for building generalizable NeRF based on point- based neural rendering. This novel paradigm outperforms all image-based benchmarks and yields state-of-the-an performance.
We explicitly model the visibilities by geometric priors and augment it with neural features, which are then used to guide the feature fetching procedure to better handle occlusions.
We propose novel nonuniform log sampling strategy based on the point density prior, and we impose perturbations to sampling parameters for robustness, which not only improve the reconstructed geometry but also accelerate the rendering speed.
We present the spatially feature- auggeeted learnable kernel as feature aggregators, which is effective for generalizability and geometry reconstruction at shape-varying areas.
2 RELATED WORK
Neural Scene Representations. Different from traditional methods that directly optimize explicit 3D geometries, such mesh Liu et al. 22200), voxe|Sitzmann al. (2019). and point cloud al. (2019), recently the use of neural networks for representing the shape and appearance scenes Sitzmann al. (2019) : Xu et al. (2019); Tancik al. (2020); Jiang al. (2020) is prevailing. More recently, NeRFMildenhal et al. (002 achieved impressive results. The following works improve NeRF in different aspects Yang et al. (2022b); Deng al. (022)). Fridovich-Kai et al. (2022) store neural features in voxels. Xu et al. (2022) integrate point cloud into neural rendering. Yariv et al. (202T) and Wang et al. (202Tb) apply NeRF model signed distance fields. 3D Gaussian splatting Kerbl et al. (2023) achieves fast and high-quality reconstruction for per-scene settings.
Generalizable Neural Field. Typical NeRF requires per-scene optimization, and cannot be gen- eralized to unseen scenes. In recent years. many methods Chen et al.) (202T): Johari et al.) (2022) have proposed generalizable neural fields from multi-view images. Nixe Yu et al. (2021) and
IBRNet Wang et al.) (2021c) regard multiple source views as conditions, query features from them and perform neural interpolation. MVSNeRF et al. (2021), JohharP[Johari et al. (0022) and
ENeRF Lin et al. (2022) incorporate Multiview stereo into NeRF to generate neural cost volume to encode the scene. To avoid occlusion in stereo matching, NeuralRay Liu et al. (202) infers occlu- sions in learnable fashion. The above methods either require source views to reconstruct the neural volume before each rendering or are fully inaccessible. We propose the generalizable point-based paradigm to avoid issues that can hardly be solved by image-based methods.
2

---

Published as a conference paper at ICLR 2024
(a) Feature Extraction
Robust density-guided Log Sampling
Query point
Nigh
Eq.6 Perturhations
Target View
Goonnerry Decoder
Inpat Views
Fi
Appearance Decader
(b) Visibilit- Feature Fetching
Estimnate Distance
Volame Beadering
Eq.1
Eq.
Eq. Duy
Eq. 7-17
Fags
Density
W,
Da
Eq.
Dinazz
Score
0.1
0.1
F
C
Ray dir
Fetch features by Eq.5
(d) Learnable Kermel
Figure 2: The Overview pipeline with our model. (a) depicts the hicrarchical feature extraction.
(b) is visibility-orienteed feature fetching. (c) denotes the density-guided robust log sampling. (d) illustrates feature aggregation by the feature-augmented learnable kernel.
2022; Jiang et al. 2022) combine typical NeRF with different explicit representations. CageNeRF (Peng et al.) incorporate control- lable bounding cages into NeRF. Xiang et al. (202T); Yang et al.) (2022a): Kuang et al. (2022) decom- pose appearance editing of NeRF. Manipulation is by-product of the GPF that be manipulated by users in both the appearance and geometry individually.
Editing NeRF. Most of the works (Yuan et al. 2022; Sun et al.
Point-based Rendering Point-based rendering involves generating images from point clouds. In the past years, neural-based point renderers Rusu & Cousins) (2011) Debevec et al. (1998) have made significant advancements in generating images from point clouds. Very recently, studies Dai et al. (2020); Rückert et al. (2022) have combined it with the prevailing NeRF pipeline to achieve better results, such as PointNeRF Xu et al. (2022) and Point2Pix Hu et al. (2023). Our method is partially similar to this category, while with good generalizability and performance.
METHODOLOGY
Given multi-view images, our aim to generate representations for different source scenes without per-scene optimization. For each scene, first, we initialize the point scaffold by the MVS technique. Next, we propose to explicitly compute the physical visibilities of points to each source image. Then, we fetch image features for all points oriented by the visibility scores to mitigate occlusion effects. Up to this step, the occlusion-awe aware neural point representation is built from multi-view images. We access this representation by volume rendering to synthesize novel views. Furthermore, we propose a robust log sampling strategy based on the physical visibility scores, which improves not only the rendering speed but also the quality of reconstructed geometries. Last we propose to aggregate neural features for query points via spatially learnable kernel function to obtain better generalizability. The whole paradigm described in Fig.
3.1 IISIBIITY-ORIENTIE FEATURE FETCHING
First, we extract multi-level image features from all input source views {/) via a symmetric U- Net-like Convolutional Network. As described Fig. 2 the multi-view images are first fed to the UNet-like extractor produce low (Fi R xxx88 and high (F), E R'1/4xu/4xx32 scale features, respectively. Meanwhile, the PatchmatchMVS is used to initialize point cloud scaffolds.
Moreover, we fetch these features to the point cloud based on their physical visibilities. This necessary to mitigate the adverse effect of stereo-matching inconsistency caused by occlusions. We first estimate the visible depth map for each camera viewpoint. We do not use either the depth map obtained by the MVS network directly project points into view planes by camera parameters. Because the former contains lots of noise and the latter only produces sparse depth maps with a mass of holes. Instead, inspired by volumetric rendering, we propose novel method to extract clean and dense depth prior from the point cloud. In detail, we emit rays from all pixels and sample 128 query points along each ray. We search neighbors in their adjacent areas with the radius r and
3

---

Published as a conference paper at ICLR 2024
filter out queries with no neighbors. Then we define the spatial density of each query point as the function their neighbors, in the following:
1 Eexpl d N
expl 1 2 -0)))))))
where refers to the number neighbors and zo is the coordinates of centers. The transmittance can be defined as
Tn exp
2d.r)
where d, is the density and is the search radius. As to the above, the opacity weight of each query point is determined w, T,d,. The depth can be estimated softly:
N
D= ΣV
Σ W
W
(3)
Next, the physical visibility score can be calculated by Eq. 4 where Ps denotes the value of the camera coordinates and D, can be considered as the visible depth estimation, especially efficient when the number of points causes bleeding problems via rasterization. Details are in the Appendix. is natural for Pz to be inherently smaller than its corresponding D;, zy by the definition of depth projection. Therefore, The visibility score is naturally constrained between 0 and 1.
Score 1 P. Di,ay P=
(4)
Each point only fetches features from images with the top k visibility scores by Fi,. E if ft,,, Fr.n Σi= w * fh.s and Fe... = =E w Is, where wis are computed over the top visibility scores via two independent learnable modules conditioned by the physical visibilities and features, Fi, Fb, Fe are low-level features, high-level features, and colors for each point.
F Fú, Fy = ((ff,,ie,.)) fn.sE [E[[LA) UsE(1.K)
(5)
The Ω depicts the entire feature fetching process, which is also determined by their visibility scores, moreover, we augment the process with neural features, detailed description is in the Appendix.
3.2 ROBUST DENSITY-GUIDED LOG SAMPLING STRATEGY
Conventional sampling strategy either skips surfaces or requires more sampled points hence improv- ing computational burden, which not only slows the rendering but also severely affects reconstructed geometries. Point clouds contain rich geometric priors that can be used to improve the sampling strategy. We reutilize the visible depth map introduced in Sec. 3.1 to determine the intersection points of ray with the surface, which we call central points. In contrast to the uniform sampling used in NeRF, we sample points along a ray nonuniformly and irregularly by spreading out from the central point, following the guidance of Eq. T
fp] 2N
Pa + base Nx - (1-1)
(6)
in which the total number of sampled points on a ray is 2N The P. refers the central point, and the base represents the logarithm base value that controls the sparsity of the sampling process. This equation describes that we sample points on both sides of the center in a symmetric and nonuniform way. The implication of this sampling strategy is to sample more points near the surface with rich geometric information and sample fewer points far away from it. In addition, alleviate the influence of the error between the estimated and the real depths, small perturbations 5 and are added to base and P... The P. and in Eq. 6 are replaced by P. P. + ô and báse base + respectively. This is also able to avoid model trapping in the local minimum. Moreover, previous generalizable NeRF methods require two identical models to implement "coarse fine" sampling, but our model generates good sampling directly from the geometry prior and does not need the "coarse stage". Hence, consume fewer memories and less time.
4

---

Published as a conference paper at ICLR 2024
3.3 POINT-BASER NEURAL RENDERING WITH -UAUURE-AGGMETEE LEARNABLE KERNELS In the above two subsections, we discuss building the neural augmented point scaffold and the log sampling strategy. In this section, we introduce how to convert information from the neural scaffold to the sampled points. First all, we search K nearest neural points for each query point within fixed radius r, and filter out those without neighbors. Popular aggregation approaches in point- based rendering include inverse distance weighting, feature encoding, and predefined radial basis functions. They suffer from distortions and blurs when generalizing varying geometries. Because geometries and scene scales drastically vary across different scenes. Thus bridging neural scaffolds and the continuous field via simple predefined functions is challenging. Therefore, we propose feature-augmen augmented learnable kernel to aggregate features. For the nearest K points {p:...p' of the query Ps. we first compute their spatial information encoding including the distances, positions under camera coordinate systems, and relative coordinates by Eq. 7 detailed in Appendix.
M LPeat(pp p. Pt, lIp - -Pilli)
Then we input the local spatial features and their neural features (F. in Eq. the feature-augmented learnable kernel, which is illustrated by Eq. Eq. 10
contains 1, h, c) to
S = MLP(h
(8)
" H Sigmoid ReLU(HH) MLP(caa(hh,,, Pt))
K
Σ
so softmar(o * n
(10)
where eal is the concatenation operation, the spatial features are first initialized as weights wf- Mcanwhile, they are then concatenated with the neural features to produce another feature vector H. and a tunming coefficient. The tuning coefficients are multiplied with the w to adjust their ratio, and then H. weighted summation to obtain the final feature vector for each query point. The results are interpreted the concatenation of Fi and Fe in Eq. [ In contrast, if the target is a, it should be Fn. With this feature- augmented learnable kernel, our network expresses local features better and performs well at geometric discontinuities, such as the edge of an object.
or a by decoders. Besides, if the final target is the color, F. denotes
Finally, conventional volume rendering (c Σx Tj(1 exp(-or0;))c) is applied to obtain color at pixels, where T; = expp( E ,,))). The only loss we used to train our model the MSE loss between predictions and groundtruth, which is depicted as L,- = [ ΣXER l|ē, Cre(r)]
3.4 HIERARCHICAL FINETUNING
We present hierarchical finetuning paradigm. Different from previous image- generic NeRF approaches, we remove the image feature extractor before fintuning, and only maintain features attached to the point scaffold, because our method can render views independently of source images.
The proposed hierarchical finetuning paradigm includes three stages: 1. feature finetuming 2. point growing and pruning 3. point refinement. The first stage optimizes neural features stored in the point scaffold and the weights of the Iearnable kernels. The initialized point clouds sometimes are imperfect and contain artifacts and pinholes. Inspired by PointNeRF, we introduce point completion and pruning techniques in the second finetuning stage. For point completion, we grow new points at places with higher opacity (a = 1 - exp( -0, )). the opacity of a candidate point is larger than preset threshold > Topacity and the distance to its nearest neural points is not smaller than another threshold dmin 2 Taist, a new point with averaged features over its neighbors is added.
Our point pruning is different from PointNeRF because our representation does not rely on the per- point probabilistic confidence. Instead, we follow the nature-augmented kernel to recompute the opacity at the positions of neural points and delete them when they have smaller opacity.
The third stage is called point refinement. Point positions might be suboptimal in their vicinities. In this stage, we freeze features and all networks and iteratively adjust the coordinates of the point scaffold, following Eq. []] In detail, a trainable offset Ap, is assigned to each point, and u refers to the camera viewpoint. The offset is regularized with its 1.2 loss to ensure that they would not move
5

---

Published as a conference paper at ICLR 2024
Training Setting
NcRF Synthetic
DTU
BlendedMVS
Methods
PSNRT
SSIMT
LPIPSL
PSNRT
SSIM
LPIPS
PSNRT
SSIMT
LPIPS_
IBRNet
26,91
0.925
0.113
25.17
0.902
0.181
22.01
0.813
0.171
MVSNcRF
23.65
0,827
0.181
23.50
0.818
0.314
20.27
0.795
0.290
Generalization
ENeRF
26.69
0.947
0.097
25.77
0.913
0.168
21.88
0.758
0.176
Neuray
27.07
0.935
25.94
0.925
0.122
23.24
0.878
0.106
Ours
29.31
0.960
0.081
27.67
0.945
0.118
26.65
0.878
0.098
IBRNet
29.95
0.934
0.079
28.91
0.908
0.130
25.01
0.721
0.277
MVSNeRF
28.69
0,905
0,160
26.41
0.881
0.274
21.11
0.796
0.279
Finetuning
ENcRF
29.21
0,931
0.077
28.13
0,931
0.101
24.81
0.863
0.201
Neuray
30.26
0,969
0.055
29.15
0.935
0.104
26.71
0.885
0,083
Ours
33.28
0.983
0.037
31.65
0.969
0.081
28.78
0.944
0.073
Table I: Quatatitive comparisons on the three datasets under generalization and finetuning settings. The PSNR, SSIM, and LPIPS computed.
far from their original positions. The three stages experience iterative optimizations one after another until the loss no longer decreases.
N
pi argmin
Σ [|o((llp + Ap:) Igell +
||pp)l
(11)
4 EXPERIMENT
Datasets. We pretrain our model on the train set of DTU Yao et al. 20200), in which we follow the train-test split setting introduced in MVSNeRF. To evaluate the generalization ability of our model, we test the pretrained model on NeRF Synthetic Dataset Mildenhall et al. (202)), the test set in DTU Dataset and large-scale scale scenes BlendedMVS Yao et (2020).
NeRF Synthetic
DTU
BlendecMVS
Ground Truth
IBRNet
MVSNeRF
ENeRF
Ours
Figure 3: Qualitative comparisons of novel view synthesis under generalization setting.
4.1 COMPARISON WITH BASELINES
We compare the proposed method with IBRNet, MVSNeRF, and ENeRF, all of which are recent state-of-the-art open-source generic radiance field methods, in both the generalization setting and finetuning setting. All generalization methods are pretrained on the same training scene. The qual- itative and quantitative results reported in Table [ and Fig. and 4 respectively. In Table I we list the metrics of PSNR, SSIM, and LPIPS on the three datasets. Our method outperforms all other methods in both generalization and finetuning settings by considerable margin. Fig. vividly

---

Published as a conference paper at ICLR 2024
NeRF Synthetic
DTU
BlendedMVS
Ground Truth
IBRNet
MVSNeRF
ENeRF
Ours
Figure 4: Qualitative comparisons of novel view synthesis under finetuning setting.
Training Setting
NeRF Synthetic
DTU
Methods
PSNR1
SSIM
LPIPS.
PSNRT
SSIM
LPIPS.
PointNeRF
30.71
0.961
0.081
28.43
0.929
0.183
Finetuning
Point2Pix
25.62
0.915
0.133
24.81
0.894
0.209
Ours
33.28
0.983
0.037
31.65
0.970
0.081
Table Quantitative comparisons with point-based methods on NeRF Synthetic and DTU datasets.
depicts that other models produce severe artifacts places with local shape variations, especially edges of the object. Existing methods cannot handle extremely varying local shapes because they only consider independent sampled points. Instead, our model fully utilizes the geometric prior and presents the learnable kernels to better abstract local features.
In the finetuning setting (Fig- 4. it is observed from DTU results that the other three models in- correctly infer the geometry, caused by occlusions in source views. Furthermore, in the Blended- MVS example, the other three models completely collapsed. This is because the source images distributed in scattered manner, and occlusions exist between adjacent views, which damage the image-based methods. By contrast, our model leverages visibility fectively address the issue. Even if the novel view lacks enough nearest souree views, model can generate plausible high-quality images. We also show the quantitative comparison in Table between approach with other paint-based rendering methods, PointNeRF Xu al. (2022) and Point2Pix Hu et al. (2023) (Point2pix is our unofficial implementation). The results show our method still has significant ad advantages. More detailed comparisons are depicted in the Appendix.
feature fetching to ef-
4.2 COMPARISON WITH NEURAY
We individually report the comparisons with Neuray in this subsection because it implicitly models the occlusion by a trainable visibility module. However, it performs poorly in out-of dittrf-distriuution scenarios. In contrast, we compute the visibility in an explicit way and slightly adjust it based on neural features. Our method yields better geometries and consistencies between views, which can be seen Fig. 5 It is observed that NeuRay generates blur and unsharp geometries, resulting in incorrect reconstructions in local areas with drastically varying shapes. Nevertheless, produces clean and sharp geometrics even though at places with geometric discontinuities. Further- more, also compute the metrics of NeuRay the three datasets, which report in Table T as
method

---

Published as a conference paper at ICLR 2024
General
General
Finetune
NeuRay
NeuRay-Depth
Ours
Ours-Depth
Figure 5: Qualitative Comparisons between ours and the implicit occlusion-awarc image-based ren- dering method: NeuRay, including the rendering views and the reconstructed geometries. The green box is the groundtruth at the right side of the figure.
Methods
Training Time
Rendering Time
PSNR f
SSIM T
LPIPS 1
Uni 64
22h
86.6s
21.42
0.776
0.312
Uni 128
26h
103.2s
25.08
0.878
0.382
Uni 64+128
29h
124s
26.56
0.903
0.179
Surl
11.5h
0.62s
24.10
0.891
0.226
log16 1n-oc log16(ours)
8.5h
1.04s
27.13
0.927
0.136
8.5h
1.04s
27.67
0.945
0.118
Ablations about different sampling strategies. Uni+number refers to uniform sample "num- ber" points. Uni 64+128 denotes the "coarse fine" strategy used in conventional NeRF, Surf2 follows the method in ENeRF, only sampling 2 points around the surface. log16-noe is the log sam- pling without perturbation. log (ogl6()urs) refers the full setting of log sampling.
Table 3:
well. Obviously, NeuRay delivers more remarkable results than the other three benchmarks thanks to its implicit occlusion modeling. However, it suffers from degeneration when generalized to out- of-distribution scenes. Besides, compared with our explicit visibility modeling, implicit modeling causes severe damage to view consistencies, which are shown in our supplementary video. By con- trast, our model gives preferable reconstruction qualities on both geometries and novel views under both generalization and finetuning settings.
4.3 ABLATION STUDIES AND ANALYSIS
Main components. This subsection presents the effectiveness of the other main components of our method, including the robust log sampling and the Prature-augmented learnable kernel. We experimentally prove the necessity of the irregular log sampling strategy in terms of performance and time by comparing it with uniform and surface sampling methods. Table 3 illlustrates that the novel sampling strategy delivers better performance while consuming less time. A visual aid provided in Sec. E of Appendix. Even though the rendering speed of Surf2 used in ENeRF is slightly less than ours, its performance is largely degraded because the estimated surface is not accurate thus sampling only 2 points around it leads dramatic errors. The metrics evaluated on the DTU test set under generalization settings, we also present visual aids to facilitate understanding in Appendix Sec. E.
Second, we compare our feature auguueree spatial learnable kernel with inverse distance weight- ing Xu al. (2022), Gaussian radial basis kernel Abou-Chakra et al. (2022), and trainable feature
8

---

Published as conference paper at ICLR 2024
Methods 7 PSNR T
Inverse distance weights
Radial basis kernel
Trainable encoding
Ours
23.61
24.75
25.20
27.67 31.65
28.18
25.88
28.92
Table 4: Quantitative Comparisons with different feature aggregators. "g" refers to generalizable setting and "ft" denotes finetuning.
2000
4000
8000
(a) growing
(b) refinement
Figure 6: Instances of point completion and refinement in finetuning.
encoding Wang et al. 2021c) for feature aggregation. Table quantitatively gives the comparison results. is noted that the inverse distance weights yield the worst quality in generalization, whereas significantly improved after finetning. The predefined kernel function produces good results in gen- cralization but does not benefit so much from finetuning. However, our method produces the best results in both generic and finetuning.
Finetune. In addition, we evaluate our hierarchical finetuning paradigm, especially in the second and third stages. It can be observed from Fig. (a). that the holes caused by imperfect initial point cloud have gradually filled up with the increasing training steps. is clear from Fig. (b) that the point refinement module enables points closer the real object surface, which effectively benefits both the
log sampling procedure and the reconstructed geometry.
(a) Color swap
(b) Shape stretch
(c) Joint twist
Figure 7: Interactive manipulation on appearance and geometry
Interactive and Editable. Our approach also comes with good by-product compared with cur- rently popular image-based generic NeRF models. Our representation is easy to interactively ma- nipulate. Here we simply conduct some preliminary attempts to show its potential for manipulation. For well-deployed neural point scaffold, manipulate the positions of points and maintain the features stored in them. The results are shown in Fig. 7 The object in scenes either move to places (a) slightly change its shape (b). Besides, it can also be handled under physical constraints (c).
open-source 3D software, such Blender, to
DISCUSSION
Conclusion. This paper proposed novel point-based paradigm for building generalizable NeRF. This model explicitly models physical visibility augmented with features to guide feature fetching, which better solves occlusions. Besides, robust log sampling strategy is proposed for improv- ing both reconstruction quality and rendering speed. Moreover, we present the spatially feature- augmented learnable kernel to replace conventional feature aggregators. Thanks to this, the perfor- mance at geometric discontinuities is improved by a large margin. Experiments on the DTU, NeRF, and BlendedMVS datasets demonstrate that our method and recover correct geometries for both generic and finetuning settings.
render high-quality plausible images
Limitation. Our GPF requires the PatchmatchMIV to initialize the point scaffold in the generaliza- tion stage. In the future, we plan to propose a neural-based initialization module that can be jointly trained with our modules simultaneously.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jad Chou-Chakra, Feras Dayoub, and Niko Sünderhauff. Particlenerf: Particle based encoding for online neural radiance fields in dynamic scenes. arXiv preprint arXiv:221 1.04441,, 2022.
Anpei Chen, Zexiang Xu, Fugiang Zhao, Xiaoshuai Zhang. Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo, Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14124-14133, 2021.
Peng Dai, Yinda Zhang, Zhuwen Li, Shuaicheng Liu, and Bing Zeng. Neural point cloud rendering via multi-plane projection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7830-7839, 2020.
Paul Debevec, Yizhou Yu, and George Borshukov. Efficient view-dependent image- based rendering with projective eextree-mppingg In Rendering Techniques' 98: Proceedings of the Eurographics Workshop in Vienna, pp. 105-116, 1998.
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 12882-12891, 2022.
Sara Fridovich-Kein Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5501-5510, 2022.
Tao Hu, Xiaogang Xu, Shu Liu, and Jiaya Jia. Point2pix: Photo-realistic point cloud rendering via neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8349-8358, 2023.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 406-413, 2014.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6001-6010, 2020.
Kaiwen Jiang, Shu-Yo Yu Chen, Feng-Lin Liu, Hongbo Fu, and Lin Gao. NeRFFaceFditing: Disentan- gled Face Editing in Neural Radiance Fields. In SIGGRAPH Asia 2022 Conference Papers, pp- 1-9, 2022.
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18365-18375, 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4))1-14, 2023.
Zhengfei Kuang, Fujun Luan, Sai Bi, Zhixin Shu, Gordon Wetzstein, and Kalyan Sunkavalli. Palet- t:NeRF: Palette-based Appearance Editing of Neural Radiance Fields. 2022.
Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022.
Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Mare Habermann, Wenping Wang. and Christian Theobalt. Neural rendering and reenactment of human actor videos. 3815):1-14, 2019.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. A general differentiable mesh renderer for image-based 3D reasoning. 44(1):00-62, 2020.
10

---

Published as a conference paper at ICLR 2024
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image- rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7824-7833, 2022.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. 65(1)999-106 2021.
Yicong Peng, Yichao Yan, Shengqi Liu, Yuhao Cheng, Shanyan Guan, Bowen Pan, Guangtao Zhai, and Xiaokang Yang. CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Defor- mation and Animation. In Advances in Neural Information Processing Systems.
Darius Rückert, Linus Franke, and Mare Stamminger. Adop: Approximate differentiable one-pixel point rendering. ACM Transactions on Graphics (ToG), -144):1-14 2022.
Radu Bogdan Rusu and Steve Cousins. 3d is here: Point cloud library (pel). In 2011 IEEE international conference on robotics and automation, pp. 1 1-4. IEEE, 2011.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2437-2448 2019.
Chunyi Sun, Yanbing Liu, Junlin Han, and Stephen Gould. NeRFEditor: Differentiable Style De- composition for Full 3D Scene Editing. 2022.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. 33:7537-7547, 2020.
Fangjinhua Wang. Silvano Galliani, Christoph Vogel, Pablo Speciale, and Mare Pollefeys. Patch- matchnet: Learned multi-view patchmatch stereo. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14194- 2021a.
Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. 2021b.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Barall- Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi- view image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4690-4699, 2021c
Fanbo Xiang, Zexiang Xu, Milos Hasan, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Hao Su. Neutex: Neural texture mapping for volumetric neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7119-7128, 2021.
Qiangeng Xu, Weiyue Wang. Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. 32, 2019.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu. Kalyan Sunkavalli, and Ul- rich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5438-5444 2022.
Bangbang Yang, Chong Bao, Junyi Zeng, Hujun Bao, Yinda Zhang, Zhaopeng Cui, and Guofeng Zhang. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. Computer ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings. Part XVI, pp. 597-614. Springer, 2022a.
Guo-Wei Yang, Wen- Ban-Yang Zhou, Hao- Yang Peng. Dun Liang, Tai-Jiang Mu, and Shi-Min Hu. Recursive-NeRF: An efficient and dynamically growing NeRF, 2022b.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-vien view stereo networks. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1790-1799, 2020.

---

Published as a conference paper at ICLR 2024
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. 34:4,05-4815, 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. Pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4578-4587, 2021.
Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. NeRF-editing: Geometry editing of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18353-18364, 2022.
12

---

Published as a conference paper at ICLR 2024
Appendix
A MODEL ARCHITECTURES
In this section, we complementally introduce the architectures of the visibiltty-oriented feature fetch- ing module and our low and high-level feature aggregators.
A.1 DETAILED DESCRIPTIONS OF THE VISIBILITY SCORES
As we explained in the main body of this paper, we first estimate the depth maps by the proposed point density-basee method. Notably, z-buffer point rasterization be used as long as the number of points is sufficient to avoid the "bleeding" problem because the geometry prior is just estimated and will be compensated by the following neural augmented module.
splatting methods can also
Then we precompute physical visibility scores cach point to all source images, as Fig. states. We use Fig. 9 to help clearly depict the physical meaning behind the design of the visibility scores. In this figure, in the camera coordinate system, P: represents the z-value of a specific point. indicates the distance between the projection of the line connecting the point and the camera center to the camera's 's optical axis. The value aay is obtained by interpolating the depth map using Eq. [ to If point, such as point A in the figure, visible from the viewpoint, it implies that it lies on the object's surface. In this case, the z-value of the point in the camera coordinate system should roughly match the interpolated depth value, Dr.zg. On the contrary, if a point (e.g., point is not visible from the viewpoint, its z-coordinate can only be greater than Di.ay. There should not be any other points between Dirry and the camera's center, as it would cause a change in the value of Dr. accordingly. Consider another case where a point lies outside the viewing angle's frustum, and its z-value is smaller than the minimum depth on the depth map, as illustrated by point C in the figure. In this case, the point would be projected outside the image plane. However, during interpolation on the plane, we employ zero padding. Therefore, the value of Diay for this point would zero, which is smaller than P. well.
Projection Routine
Neural Point
Reprojection Point
V
Visible Score
V
Ng 0.3
V
V. 0.7
Figure 8: Schemetic diagram of physical visibility score computing. We precompute visibility scores for each independent point to all source images. V, refers to the score which ranges from 1, larger score represents more possibility of this point to be viewed from the associated view.
A.2 DETAILED IMPLEMENTATION OF FEATURE AGGREGATORS
When the visibility scores are obtained, we select three source views with the first three largest visibilities for each point. Next, the features the three selected views will be aggregated by
13

---

Published as a conference paper at ICLR 2024
Ray,
Ray:
P.c
P.a
B
Viewpoint
P2.
D
Object
Plane
Padding
Figure 9: Visualization of the physical meaning of the visibility score.
our proposed attentive visibility module, instead of simply averaging over them, in a more refined manner. First, the fetched low- and high-dimensional features are projected to 32 dimensional space by MLPs. Here we design two aggregators, one for low-level features and colors, the other for high-level features. Regarding the low-level feature aggregators, multi-layer perceptron (MLP) consisting of two hidden layers is employed. This MLP takes in the concatenation of low-level features, colors, and visibilities as input and generates weights for features and colors as output (refer to Eq. [.. In the equation, the variable represents the physical visibility scores. The feature tuning weights, denoted as wh. are utilized to make slight adjustments to the original visibility scores. The visibility scores their summation equals 1, then the weighting sum of h; is performed to obtain the final features with the consideration of visibilities, stated in Eq. 13
multiplied by the feature tuning weights and then normalized to ensure that
il..... S(Fi, C, ( il......
(12)
h F = Σ Norm(w;e jEk o
(13)
When it comes aggregating the high-level features, there exists minimal disparity. The aggregator employed for high-level features also utilizes a multi-layer perceptron (MLP), albeit with distinct in- puts. Initially, features are concatenated with colors akin the low-level aggregator. Subsequently, the mean and variance are calculated across the k vectors, and the resultant values are concatenated as an additional input vector. This approach is adopted due the inclusion more comprehensive information pertaining to the scene geometries within the high-level feature consistency.
We here formulate a vivid diagram to depict the process of the feature- augueetdd learnable kernel (Eq. 8 to Eq. 10 in the main paper) in Fig. IO Clearly, the weights to sum neural features are initially produced by spatial information the above branch, but experience tuning with the feature augmentations of the below branch. Besides, the cross-scene decoders are two simple MI.s with
D P.
P
MLP
MLP
norm
sigmoid
F
MLP
Neural Features (F.)
relu
Figure 10: Schemetic diagram of the feature-augmented learnable kemel.
two layers to translate neural features aggregated from their neighboring points to the density and the color weights. Simple decoder architectures can enhance the representation ability the neural features.
14

---

Published as a conference paper at ICLR 2024
GT
IBRNet
MVSNeRF
ENeRF
Ours
Figure 11: Qualitative comparisons on LLFF dataset.
Training Setting
Generalization
Finetuning
Methods
PSNR
SSIMI
LPIPS
PSNRT
SSIMT
LPIPS
IBRNet
25.13
0.817
0.205
26.73
0.851
0.175
MVSNeRF
21.93
0.795
0.252
25.45
0.877
0.192
ENeRF
22.78
0.808
0,209
24.89
0.865
0.199
Neuray
25.35
0.818
0.198
27.06
0.850
0.172
Ours
26.01
0.829
0.184
27.79
0.872
0.171
Table 5: Quantitative comparisons LLFF dataset.
B IMPLEMENTATION DETAILS
We train our generalizable model on single RTX3090 GPU for 100k iterations using Adam op- timizer with an initial learning rate 5e-4 and a cosine annealing schedule with annealing a of 0.1. In our experiments, in the training stage, selected 10 input views to compute the visibility and set the top-k as top-3 to perform visibility- compute visibility and fetch features across all source images. In our log sampling strategy, the two perturbations are sampled from the two uniform distributions respectively. The parameters of the distributions scenes. The base is set to 1.35 and we sample 16 points for each ray. For each iteration, the training batch is 512. In the feature-augmented learnable kernel, the nearest neighbors are selected 8. Besides, we enlarged the NeRF Synthetic scene 100 times to make it in accordance with the scale of DTU training set. In finetuning, we first extract and aggregate all image features the point scaffold. Then, source images are not required in the finetuning setting. Moreover, the finetuning is effectively fast and only consumes 50k iterations.
feature fetching. In contrast, the test stage,
(-10 10) and normal ~ N (0, 0.01) determined by the scale of the
In the first stage of finetuning setting, the features and parameters of networks have the initial learn- ing rate of le-5 but the color of each point trained with 1e-7. In the point completion stage, notably, only if the radius of the hole is smaller than the searching radius, the point cloud can be re- covered. Therefore, we slightly enlarge the search areas
times than that the generalization stage. To extend searching areas, we reutilize uniform sampling. In point pruning, it is impossible to check all the neural points at each iteration step, thus we randomly select 3192 neural points at cach pruning step. After the refinement of point positions, the precomputed visible depth map should be regenerated.
C ADDITIONAL RESULTS AND ANALYSIS
In this section, we provide additional qualitative results under generalization and finetuning settings with enlarged details.
15

---

Published as a conference paper at ICLR 2024
RRBB
NeRF Synthetic
DTU
BlendedMVS
Ground Truth
IBRNet
MVSNeRF
ENeRF
Ours
Figure 12: Additional qualitative results on the three datasets with enlarged details under general- ization setting.
NeRF Synthetic
DTU
BlendedMVs
Ground Truth
IBRNet
MVSNeRF
ENeRF
Ours
Figure 13: Additional qualitative results on the three datasets with enlarged details under finetuning setting.
C.I QUALITATIVE RESULTS ON THE LLFF DATASET
To provide further comparisons and evaluations, we conducted additional experiments on the LLFF dataset for all baselines and our proposed method. This section includes the qualitative results, depicted in Fig. 11 as well as the quantitative results presented in Table 5 These findings offer insights into the performance and effectiveness of method in comparison to the baselines.
C.2 QUALITATIVE RESULTS WITH ENLARGED DETAILS
In Fig. 12 and additional results under generalization and finetuning settings with their large detail boxes are given. Our representation performs stably under all three datasets, even the ge- ometry complex the source images not close to the target view. However, in BlendedMVS
16

---

Published as a conference paper at ICLR 2024
Dataset, the camera views often distribute sparsely, thus image-based rendering cannot acquire suf- ficient information from a few surrounding source images. Utilizing all source images would cause computation redundancy. Therefore, the counterparts cannot deliver satisfactory results the three Datasets. Furthermore, even if finetuning can improve the quality of rendering images of the coun- terparts, it cannot deal with the in-realism caused by occlusion. The chair scene in NeRF Dataset is an example of that. The back leg of the chair disappears. Moreover, the mic scene in NeRF dataset contains many geometry inconsistencies, which lead blur and artifact of reconstruction images, our model performs better in such scenes well.
D COMAPRISONS OF THE RECONSTRUCTED GEOMETRY WITH BASELINES
Fig. 5 indicates the performance of our model on the reconstruction of geometries. The predicted depth map is clear and contrasting. In this section, report the quantitative results to further describe our benefits in Table 6
In this table, The RMSE refers to the root mean square enror computed over the normalized depth groundtruth, Acc. denotes the accuracy of the threshold, we set the threshold as 195% for all experiments. The OurSwo/vis refers the variant of our model by removing the visibility scores. In addition, all tests are under generalization settings because finetuning cannot faithfully reflect the real understanding of models to the realistic geometries. The performance would see decrease if we remove the explicit occlusion modeling, which proves the validity of the usibility-oriienee feature fetching module.
Method
IBRNet
ENeRF
Neuray
wotS
Ours
RMSE,
0.677
0.527
0.547
0.294
0.161
NeRF Synthetic
Acc. T1
0.380
0.159
0.119
0.596
0.787
RMSE
0.321
0.435
0.162
0.189
0.122
DTU
Acc.T1
0.896
0.741
0.911
0.905
0.936
Table 6: Quantitative comparisons of depth maps on NeRF Synthetic and DTU datasets.
E ADDITIONAL ABLATION STUDIES
Additional visualizations about ablation studies for the main components are provided in this sec- tion to further interpret Sec. 4.3 in the main paper. Besides, we report the ablations of the depth map generated by our model with different configurations. Moreover, the effect of point numbers is compared and reported. Here we first present Fig. 14 to visually help understand the diffcrences between various sampling strategies, i.e. Table in the main paper. In this figure, the x-axis cor- responds to training time, the y-axis represents the PSNR, and the size the circles indicates the rendering time for an 800*800 image. By examining the figure, can readily observe that the log sampling method consumes the least amount of training time while achieving the highest PSNR. Additionally, demonstrates the second-faste rendering speed, slightly slower than the surface sampling technique.
E. VISUALIZATIONS OF ABLATION STUDIES
Fig. 15 shows the detailed structure at places with dramatically varying geometries under gener- alization settings. Our learnable kernel reconstructs the sharpest edges with fewer blurs and fogs, whereas other methods suffer from confused appearances and disconnected shapes.
Furthermore, Fig. 16 gives visual examples to compare different sampling strategies. We can see that uniform sampling fails to maintain detailed textures in renderings, even though the number of sampled points increases to 192. However, our density-based log sampling can recover distinct appearances and unambiguous geometry.
17

---

Published as a conference paper at ICLR 2024
30
lóg16 (ours)
28
1.04s
27.67
Uni 64+128
124s
26
log16-neE
1.04s
PSNR 1
Uni 128 103.2ş
24
Surf
22
0.62s
Uni 86.6s
20
Rendering Time
10
20
30
Training Time (h)
Figure 14: Ablation studies of different sampling strategies.
Inverse distance
Radial basis function
Feature encoding
Groundtruth
Ours
Figure 15: Ablation study of various feature aggregators.
E.2 VALIDATION OF THE LOW- AND HIGH-LEVEL DECOMPOSITION
We conducted experiments on the DTU dataset to analyze the impact of different combinations of features the rendering qualities, including only low-level features, only high-level features, and the combinations of low- and high-level features. The ablation results are listed in Table Here we evaluate the rendering quality both generalization and finetuning settings and the reconstructed depth map in the generalization setting. The results show that separating appearance and shape into individual code is indeed meaningful.
Training Setting
Generalization
Finetuning
Methods
PSNRT
SSIM1
LPIPS
PSNRT
SSIM
LPIPS|
RMSE,
Acc.Tt
Only-low
24.15
0.896
0.178
30.16
0.921
0.133
0.383
0.865
Only-high
22.94
0.852
0.231
30.47
0.928
0.132
0.155
0.917
low-high
27.67
0.945
0.118
31.65
0.970
0.081
0.122
0.936
Table 7: Quantitative comparisons between the compositions of different levels of features.
E.3 RECONSTRUCTION OF GEOMETRIES
Here we qualitatively report the depth visualization under different model settings. Fig. 17 (b) refers to the groundtruth depth, and (c) is the depth predicted by PatchmatchMVS Net Wang et al. (202 la). (d) denotes our full model prediction. (e) is the prediction our model except for using the feature- augmented learnable kernel introduced in the main paper. (f) is also the prediction of our model but using "coarse to fine" uniform sampling rather than the proposed log sampling. From these figures, observe that the full model provides the most reasonable prediction. Besides, the prediction of
18

---

Published as a conference paper at ICLR 2024
a uniform64
(b) uniform128
(c) uniform 192
(d) logl6
(e) GT
Figure 16: Ablation study of different sampling strategies.
(a) Image
(b) GT
(c) MVS
(d) Full model
(e) No Learnable Kernel
(f) Uniform
Figure 17: Ablation study of depth predictions.
MVSNet is unable handle regions without depth, which also occurs in the prediction of other generie NeRF models.
PSNR/ / SSIM/ LPIPS
24.3/ 0.73 0.30
0.85/ 0.27
26.3 0..4//84//00 0.21
26.7 0.87 0.19
GT
15K
50K
100K
200K
Figure 18: Ablation studies of the influence of the number of points in the point scaffold. The figure reports the rendering quality when the number of points is about 15K, 50K, 100K, and 200K respectively. The quantitative metrics including PSNR, SSIM, and LPIPS are calculated.
E.4 INFLUENCE OF THE POINT SCAFFOLD DENSITY
We additionally analyze the effect of the different number points the scaffold. In detail, we test trained model the DTU Scan 14 scene with different levels of point downsampling under the generalization setting. shown in Fig. [8 that the quality of rendering is slightly impacted by the density of the point scaffold. However, fewer number points lead to faster rendering speed.
19

---

Published as a conference paper at ICLR 2024
Thus, this is trade-off between rendering quality and speed. We recommend that the number of points 50K 100K is enough to produce plausible novel images with satisfactory speed for the DTU dataset. This experiment demonstrates the robustness of our proposed representation, which cannot be drastically affected by the point density.
In addition, found that for the large scene, fewer number of initial points would lead to smoother images due to the lack of representation of high-frequency information. Here we give an example illustrate in Fig. 19 Obviously, the initial number of points is insufficient for the large scene at (b) in the figure. leading to the smoothness of the image. That is intuitive since larger scenes typically require more points for initialization.
ENeRF
200K (ours)
400K (ours)
GT
Figure 19: Examples to indicate fewer points result in smoother images.
F MANIPULATION OF THE REPRESENTATIONS
In this paper, we also prove that the proposed representation has the potential to interact with users. In this section, in detail introduce how to edit the generalizable point field, including moving, recoloring, and stretching, which corresponds to Fig. 7 in the main paper.
Object movement. If one wants to move object in the scene, he directly move the corre- sponding neural points. As shown Fig. 7 in the main body, the purple ball in the top right corner moved a distance. This the simplest way edit the scene. Because all information stored in the point representation.
Object recoloring. Our representation also supports recoloring. One example is exhibited in Fig. in the main paper. In this instance, We replace the red ball's color and low-level appearance features (F& and F1 in Eq. (5) with those in the left purple one while maintaining its high-level geometry features. In addition, if the number of points of the two objects is not identical, we compute the feature the second object by interpolating the neighbor points in the original object. We simply use the inverse distance weights to obtain the interpolated feature vectors, this process is shown in Eq. [4]
ΣN Ek=1wR WAfk fa(z) Ek=1
ΣN
(14)
wk
|| - Ppll
where N is the number of neighbors, in our experiments we select 8 as N. æ refers to the point in the second object. P&: denotes the neighboring points in the first object.
Object deformation. We stretch the stool in the drums scene in NeRF Synthetic Dataset, which can be seen in Fig. 7 in the main paper. To avoid nonuniform deformation of the point cloud, we first reconstruct the mesh representation from the original point cloud. For each vertex in the mesh, we assign the geometry and appearance features the mesh. After the deformation occurs in the mesh we sample points uniformly on the surface of the mesh and interpolate features from the mesh vertex. Here we obtain the deformed neural point field that can be used to render views. For the Lego example, we simply construct the mesh by connecting the points with their neighbors. Next, we twist the joint of the Lego model by Blender to get the deformed positions of points. While the geometry and appearance features remain the same as before deformation occurs. We can see that there is no obvious decrease in the rendering qualities between the original images and the deformed images.
by Eq. 14 Then we use Blender stretch
20

---

Published as a conference paper at ICLR 2024
G COMPARISON WITH OTHER POINT-BASED RENDERING METHODS.
Some existing methods combine point cloud with NeRF to conduct point-based rendering, repre- sented by PointNeRF and Point2Pix. Our approach substantially differs from other point-based rendering methods. Works represented by PointNeRF focus on improving rendering quality and speed with the assistance of point clouds. Those denoted by Point2Pix focus on rendering novel views from colorful point clouds. However, our goal to train a model that can generate neural representations from multiview images generalizing to different scenes. In this section, we compare the three approaches detail. Tgvees information about the quantitative results for comparing the three approaches.
Training Setting
NeRF Synthetic
DTU
Methods
PSNRT
SSIM
LPIPS
PSNR
SSIMT
LPIPS!
PointNeRF
6.12
0.18
0.88
23.18
0.87
0.21
Generalization
Point2Pix
19.23
0.787
0.542
16.74
0.655
0.558
Ours
29.31
0.960
0.081
27.67
0.945
0.118
PointNeRF
30.71
0.961
0.081
28.43
0,929
0.183
Finetuning
Point2Pix
25.62
0.915
0.133
24.81
0.894
0.209
Ours
33.28
0.983
0.037
31.65
0.970
0.081
Table 8: Quantitative Comparisons with point- based methods on NeRF Synthetic and DTU datasets.
G.1 COMPARISON WITH POINTNERF
Our method is novel point-based paradigm for generalizable NeRF reconstruction, whereas Point- NeRF focuses on single-scene optimization. PointNeRF also requires pertaining to the DTU dataset, while it can be seen
initialization and cannot generalize unseen scenes.
In Fig. 20 we give an example to illustrate this. From this figure and Table g PointNeRF performs well on the DTU dataset but collapses the NeRF dataset due to its pre-training the DTU dataset. This lack of generalization ability is significant drawback for PointNeRF. In contrast, our proposed model, although also trained the DTU dataset, exhibits strong generalization capability on out- of-distribution data. This ability to generalize well to unseen data is a key advantage of our method.
Additionally, we note that Point2pix, another baseline model, tends produce blurry images. This limitation hampers the visual quality and fidelity of the rendered images.
bosnt-based rendering methods can generally be divided into two main steps: "image-to-point fea- ture fetching" and "point-to-ray feature aggregation." We will outline their distinctions in terms of the two steps.
In the "image to point" step:
I.PoinNeRRF fetches features for points solely from single image, but our method considers multiple-view images simultaneously, incorporating a broader range of visual information. 2. Addi- tionally, our method explicitly takes occlusions into account during feature fetching. Furthermore, while PointNeRF employs a single latent vector as the neural feature for each point, our method utilizes pair of latent vectors to separately represent shape and appearance. 4. Our method lever- ages level features to capture color information and high-level features for geometry regression, PointNeRF only utilizes high-level features.
In the "point to ray" step:
1. PointNeRF employs uniform sampling to sample points, whereas our method fully utilizes ge- ometry priors in the point cloud by employing a log sampling strategy. This enables our method to render more efficiently and achieve superior geometries. 2. PointNeRF simply aggregates features from the neural point cloud to the sampling points along rays using inverse distance weights. In trast, our method introduces a learnable kernel that enables feature aggregation based on visibilities, improving the effectiveness of the aggregation process.
21

---

Published as a conference paper at ICLR 2024
(a) PointNeRF
(b) ours
(c) PointNeRF-ft
(d) ours-ft
(e) GT
Figure 20: Comparisons with PointNeRF. ft refers to the per-scene optimization results.
Point2Pix
Ours
GT
Figure 21: Comparison with Point2Pix.
G.2 COMPARISON WITH POINT2PIX
Point2pix is proposed to render novel views from colorful point clouds rather than reconstructing neural radiance fields from multiview images, which different from our methods. However, can be categorized as point-based rendering. Therefore we also compare our method with it in terms of rendering qualities. We reimplement Point2pix with Pytorch and test it under our dataset configurations because there is no source code available at this moment. The qualitative results are shown in Fig 21 The point2pix cannot correctly render views at scenes with complex geometries and locally shape-varying areas. In addition, it is adversely affected by the imperfect point cloud, resulting in large holes and distortions renderings. However, thanks to the point completion and finetuning in hierarchical finetuning scheme,
faithfully fill up holes and iteratively refine point clouds to achieve much better reconstructions. Morcover, our log sampling and learnable kernel modules enable perform well at geometric discontinuities.
22


Reviews:
[{"review_id": "XKGtDO1SU9", "reviewer": "Reviewer_BFzR", "summary": "This paper proposes a novel paradigm for constructing a generalizable neural field based on point-based rendering, which addresses the challenges of occlusions, distortions, and degradation in image-based representations. The proposed approach combines geometric priors and neural features to eliminate occlusions in feature-fetching, and a nonuniform log sampling strategy and a learnable kernel spatially augmented with features for improved rendering speed and reconstruction quality. The authors demonstrate the effectiveness of their approach on a variety of datasets, showing improved generalization and robustness to occlusions and distortions compared to previous methods. Overall, the paper presents a promising approach for learning robust and generalizable radiance fields.", "questions": "1. Visualize the test PSNR curve instead of just stating \"Convergence Speed\" as in Table 2 would be more convincing, intuitive and easier to follow.\n2. Some modules are not well ablated and analyzed - \\eg high-low-level feature encoding.\n3. It would be nice to present some metric and quantitatively prove the quality in geometry (esp. occlusion awareness). At least depth error should be compared with MVSNeRF and NeuRay.\n4. In related works, Generalizable Neural Field. section: \"All the above can be seen as image-based neural rendering\". I think this might be inaccurate- I believe the finetuned/unfinetuned MVSNeRF / GeoNeRF / NeRFusion can aggregate multi view information and do not require original images for further use (though MVSNeRF fetches image color for rendering in some versions). Could you clarify on this? Also I believe the section is not extensive enough. The authors should also talk specifically about other point-based neural rendering methods, maybe in a dedicated section.\n5. Maybe: considering other point-based methods as ft baselines and include in the main paper.\n6. Typos and minor fixes:\n  - Table 2: Convergeuce -> Convergence; Missing/misplaced underline under 1.04s\n  - Citation format should be fixed throughout the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall the paper is nicely presented and introduce several novel components including: \n\n- The proposed approach combines geometric priors and neural features to eliminate occlusions in feature-fetching explicitly in the rendering process, which is a novel contribution to the field.\n- The authors introduce a nonuniform log sampling strategy and a learnable kernel spatially augmented with features, which is a novel approach to improving rendering speed and reconstruction quality.\n\nThe method is properly evaluated with some ablation study.", "weaknesses": "The paper proposed a new pipeline with several novel designs over the existing methods, however many of the designs are not validated and some of the claims are not strongly backed by their existing experiments:\n1. The exact definition of convergence speed is vague in Table 2, and is not explained in details. Making the results in this table questionable.\n2. Separating low level and high level features sounds intuitive but however is not validated and the necessity of such design is thus questionable.\n3. No quantitative validation on claims such as \"better geometry\" and \"occlusion awareness\".\n\nAlso the paper needs more work to reduce typos and fix the format.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TYmh6q4vPX", "reviewer": "Reviewer_BQBs", "summary": "The work propose a generalizable point-based NeRF for novel-view synthesis tasks. The point cloud is first initalized from classical MVS technique. A U-Net is trained to extract feature for the input views, which is aggregated into the points in a visibility-aware and learnable manner. To do volume rendering, query coordinates are sampled per the point density to improve efficiency. The query coordinates search K-nearest neighbor from the feature point cloud to form the query point feature, which is then map to density and color for volume rendering.", "questions": "I found some qualitative results from the baseline is better. In Fig3, the head of the ship and the sea in the ship scene, the table of the durian scene are better recovered by ENeRF. In Fig4, the reconstructed ground dirt in the BlendedMVS is overly smooth while the baseline ENeRF can recover more detail texture. What would be the reason of these? Is it because the dependent MVS point cloud? \n\nIs the proposed method sensitive to the initial points?\n\nPaper proofread:\n- Missing parentheses for the $\\exp$ in Sec3.3 last paragraph.\n- The reference to the Figure in Sec.4.2's 5th sentence is missing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The quantitative improvement is solid. I believe many of the proposed modules can be plug into point-based rendering system to boost results.", "weaknesses": "The visibility score in Eq4 is not well designed. The score actually decay more quickly for point ahead of the depth ($P_z < D_{i,xy}$). Consider two points with $P_z^{(back)} = D_{i,xy} + \\epsilon$ and $P_z^{(front)} = D_{i,xy} - \\epsilon$, their scores are:\n- $score^{(back)} = 1 - \\frac{|D_{i,xy} + \\epsilon - D_{i,xy}|}{D_{i,xy} + \\epsilon} = \\frac{D_{i,xy}}{D_{i,xy} + \\epsilon}$\n- $score^{(front)} = 1 - \\frac{|D_{i,xy} - \\epsilon - D_{i,xy}|}{D_{i,xy} - \\epsilon} = \\frac{D_{i,xy} - 2\\epsilon}{D_{i,xy} - \\epsilon}$\n\nWhen $\\epsilon > 0$, $score^{(front)} < score^{(back)}$. In addition, the score is claimed to be naturally constrainted in range of 0 and 1, but it is not the case when $P_z < 0.5 D_{i,xy}$ (become negative).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6lJsAb8vX8", "reviewer": "Reviewer_s3fi", "summary": "The paper presents a novel paradigm in generalizable Neural Radiance Field (NeRF) research by introducing the Generalizable Neural Point Field (GPF). Unlike traditional NeRF methods that utilize image-based rendering, this work focuses on point-based rendering. The paper claims to address several prevalent issues with the existing image-based methods, including occlusion-related problems, artifacts, and performance drop-offs with varying view distances.", "questions": "- Could the authors expand on the visibility-oriented feature fetching, possibly with diagrams, for better clarity? Figure (b) alone is not clear enough.\n- Given the reliance on PatchmatchMVS for the initial point scaffold, how does this affect the scalability or deployment of GPF in diverse scenarios? The authors also mention in the limitation section that they want to propose a NeRF-based initialization module that can be trained from scratch. Please comment on how you plan to achieve this.\n- Can the authors comment on potential efficiency challenges due to the method's complexity? For example, compare the rendering speed with E-NeRF.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Originality: The Generalizable Neural Point Field (GPF) is a fresh perspective in NeRF research, emphasizing point-based over image-based rendering.\n- Quality: The proposed methods exhibit high-quality research and innovation, from the nonuniform log sampling strategy to the feature-augmented learnable kernel.\n- Clarity: The paper's overall structure is clear, presenting a logical flow of ideas and discussions.\n- Significance: If the claims are validated further, this research could serve as a benchmark in the NeRF domain.", "weaknesses": "- Lack of Detailed Explanations: Some sections, especially the technical components, could use more in-depth explanations or visual aids.\n- Dependency on Other Technologies: The initial dependency on PatchmatchMVS might limit the paper's approach from being a standalone solution.\n- Limited Experimentation: Testing on only three datasets might not showcase the full potential or limitations of the method.\n- Complexity: The approach's intricate nature might pose scalability or efficiency challenges that haven't been addressed comprehensively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GjTCfyCjZr", "reviewer": "Reviewer_gQRV", "summary": "This paper proposes to construct the generalizable neural field, called the generalizable neural Point Field (GPF), based on point-based rendering. This approach explicitly models by geometric priors and augments it with neural features to eliminate occlusions in feature-fetching. A nonuniform log sampling strategy is proposed to improve both rendering speed and reconstruction quality. Moreover, this paper presents a learnable kernel spatially augmented with features for feature aggregations, mitigating distortions at places with drastically varying geometries. Experiments show that the proposed model can deliver better geometries, view consistencies, and rendering quality on three datasets in both generalization and finetuning settings.", "questions": "Please clarify the comparison with PointNeRF and explain the key factors that make this method superior to PointNeRF.\n\nPlease discuss the comparison between this method and Gaussian-splatting.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a Generalizable neural Point Field (GPF) for building generalizable NeRF based on point-based neural rendering. This paradigm outperforms existing image-based benchmarks and yields state-of-the-art performance on generic reconstructions.\n\nThis method explicitly models the visibilities by geometric priors and augments it with neural features, which are then used to guide the feature fetching procedure to better handle occlusions.\n\nA nonuniform log sampling strategy is proposed based on the point density prior, and perturbations to sampling parameters are imposed for robustness, which not only improves the reconstructed geometry but also accelerates the rendering speed.\n\nA spatially feature-augmented learnable kernel as feature aggregators is presented, which is proven to be effective for generic abilities and geometry reconstruction at drastically shape-varying areas.", "weaknesses": "This reviewer has the following concerns.\n\nThe primary contribution claimed by this paper is the introduction of the first generalizable NeRF based on point-based neural rendering. An existing point-based method is PointNeRF, which is a per-scene optimization method.\nSection F.1 discusses the comparison between GPF and PointNeRF, which is helpful. What does the entry \"Ours\" represent in the table below Figure 17? If it refers to the model after fine-tuning, what are the results without fine-tuning?\n\nUpon closer examination of the comparison with PointNeRF, this paper states that the improvement is attributed to hierarchical fine-tuning strategies. I am wondering about the runtime required for fine-tuning. PointNeRF can be optimized from scratch in approximately 40 minutes. Additionally, is pretraining of the generalizable NeRF necessary? It seems that the primary advantage of this method over PointNeRF lies in its superior fine-tuning strategy.\n\nThe log sampling strategy is simply a handcrafted sampling distribution around the surface, which may not be considered a significant technical contribution. This strategy can only be applied if the depth prior is known.\n\nIt is important to discuss the comparison between the proposed method and the recently introduced efficient Gaussian-splatting representation. What are the advantages of the proposed method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)