SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Point Feature Renderer To obtain the final rendered colour output image  $\hat{\mathbf{I}}$  for a given camera pose  $\mathcal{C}$ , we input the aggregated feature map to a convolutional feature renderer,  $\hat{\mathbf{I}} = f_{\theta_{\mathcal{R}}} (F_{\mathcal{C}})$ . Our renderer is based on a modified version of the U-Net architecture [34], featuring two downsampling layers and two upsampling layers. Notably, we remove the BatchNorm layers in this network. For more details regarding the model architecture, please refer to the appendix.
3.4 Progressive Refinement of Scene Representation
Point Pruning To eliminate potential outliers during the point position optimization process, we introduce a point pruning strategy that utilizes a learnable influence score  $\tau_{i} \in \mathbb{R}$  associated with each point. We calculate the probability  $w_{b,j} \in [0,1]$  of a ray  $\mathbf{r}_j$  hitting the background by comparing a fixed background token  $b$  to the product of each point's influence score  $\tau_{i}$  and its raw attention score  $a_{i,j}$  (defined in Eqn. 6). The probability is given by:
$$
w _ {b, j} = \frac {\exp (b)}{\exp (b) + \sum_ {m = 1} ^ {K} \exp \left(a _ {m , j} \cdot \tau_ {m}\right)} \tag {7}
$$
By collecting the background probabilities  $w_{b,j}$  for all  $H \times W$  rays cast from camera pose  $\mathcal{C}$ , we obtain a background probability mask  $P_{\mathcal{C}} \in \mathbb{R}^{H \times W \times 1}$ . The rendered output image  $\hat{\mathbf{I}}$  can then be computed using this background probability mask as follows:
$$
\mathbf {I} = \left(1. - P _ {\mathcal {C}}\right) \cdot f _ {\theta_ {\mathcal {R}}} \left(F _ {\mathcal {C}}\right) + P _ {\mathcal {C}} \cdot \mathbf {I} _ {b} \tag {8}
$$
Here,  $\mathbf{I}_b\in \mathbb{R}^{H\times W\times 3}$  represents the given background colour. All influence scores are initialized to zero. Starting from iteration 10,000, we prune points with  $\tau_{i} < 0$  every 500 iterations. Additional details and comparisons to the model without pruning are available in the appendix.
Point Growing As mentioned in Sec. 3.1, our scene representation allows for increased modelling capacity by adding more points. To achieve this, we propose a point growing strategy that incrementally adds points to the sparser regions of the point cloud every 500 iterations until the desired total number of points is reached. More information on how the sparse regions are identified can be found in the appendix. Fig. 1a illustrates the effectiveness of our point growing strategy in progressively refining the point cloud, resulting in a more detailed representation of the scene.
3.5 Training Details
Our training objective is minimizing the distance metric  $d(\cdot, \cdot)$  from the rendered image  $\hat{\mathbf{I}}$  to the corresponding ground truth image  $\mathbf{I}_{gt}$ . We define the distance metric as a weighted combination of mean squared error (MSE) and the LPIPS metric [54]. The loss function is formulated as:
$$
\mathcal {L} = d (\hat {\mathbf {I}}, \mathbf {I} _ {g t}) = \operatorname {M S E} (\hat {\mathbf {I}}, \mathbf {I} _ {g t}) + \lambda \cdot \operatorname {L P I P S} (\hat {\mathbf {I}}, \mathbf {I} _ {g t}) \tag {9}
$$
We set the weight  $\lambda$  to 0.01 for all experiments. During training, we jointly optimize all model parameters, including  $\mathbf{p}_i$ ,  $\mathbf{u}_i$ ,  $\tau_i$ ,  $\theta_K$ ,  $\theta_V$ ,  $\theta_Q$  and  $\theta_{\mathcal{R}}$ . We train our model using Adam optimizer [12] on a single NVIDIA A100 GPU.
3.6 Learning Point Positions From Scratch
We demonstrate the effectiveness of our method in learning point positions from scratch using the Lego scene in the NeRF Synthetic Dataset [22]. We compare our proposed method to recent point-based methods, namely DPBRF [53], Point-NeRF [48], NPLF [24] and Gaussian Splatting [11]. It is worth noting that NPLF does not inherently support point position learning, so we made modifications to their official code to enable point position optimization.
Point-NeRF [48]
NPLF [24]
DPBRF [53]
PAPR (Ours)
Gaussian Splitting [11]
Reference Image
Figure 4: Comparison of our method and prior point-based methods on learning point cloud positions from scratch. We modify NPLF [24]'s official implementation to support point position updates. All methods start with the same initial point cloud of 1,000 points on a sphere. Point pruning and growing strategies are disabled for a fair comparison of geometry learnability. An RGB image of the scene from the same view point as the point cloud visualizations is provided as a reference. Point-NeRF [48] and NPLF [24] fail to reconstruct a reasonable point cloud. DPBRF [53] captures a rough shape but introduces significant noise and struggles to capture the rear end of the Lego bulldozer effectively. Gaussian Splatting [11] captures the Lego silhouette but lacks structural details, such as the bulldozer's track (the black belt around the wheels). In contrast, our method successfully learns a point cloud that accurately represents the object's surface and captures detailed scene structures.
In our comparison, all methods start with the same set of 1,000 points initialized on a sphere. To ensure a fair evaluation of point cloud position learnability, we disable point pruning and growing techniques for all methods. As shown in Figure 4, our method successfully deforms the initial point cloud to correctly represent the target geometry. In contrast, the baselines either fail to recover the geometry, produce noisy results, or lack structural details in the learnt geometry.
To validate our contribution of learning point-based scene representation and rendering pipeline directly from multi-view images, we compare our method to recent point-based neural rendering methods, namely NPLF [24], DPBRF [53], SNP [56], Point-NeRF [48] and Gaussian Splatting [11]. As a secondary comparison, we demonstrate the potential broader impact of our method by comparing it to the widely used implicit volumetric representation method NeRF [22].
To assess the performance of scene modelling with a parsimonious representation, we conduct evaluations using a total of 30,000 points for each method. For all point-based baselines, we use their original point cloud initialization methods, which are based on Multi-View Stereo (MVS), Structure-from-Motion (SfM) or visual hull. Conversely, for our approach, we adopt a random point cloud initialization strategy within a predefined volume, introducing an additional level of complexity and challenge to our method. We set the parameter  $K = 20$  for selecting the top nearest points, and the point feature vector dimension  $h = 64$ . We evaluate all methods in both synthetic and real-world scenarios. For the synthetic setting, we choose the NeRF Synthetic dataset [22], while for the real-world setting, we use the Tanks & Temples [13] subset, following the same data pre-processing steps as in [48]. We evaluate rendered image quality using PSNR, SSIM and LPIPS [54] metrics.
Table 1 shows the average image quality metric scores. PAPR consistently outperforms the baselines across all metrics in both synthetic and real-world settings, without relying on specific initialization. These results demonstrate PAPR's ability to render highly realistic images that accurately capture details using an efficient and parsimonious representation.
NeRF Synthetic	Tank & Temples
PSNR ↑	SSIM ↑	LPIPSvgg ↓	PSNR ↑	SSIM ↑	LPIPSvgg ↓	Initialization
NeRF [22]	31.00	0.947	0.081	27.94	0.904	0.168	-
NPLF [24]	18.36	0.780	0.213	21.19	0.761	0.240	MVS
DPBRF [53]	25.61	0.884	0.138	17.25	0.634	0.301	Visual Hull
SNP [56]	26.00	0.914	0.110	26.39	0.894	0.160	MVS
Point-NeRF [48]	25.93	0.923	0.129	24.75	0.890	0.184	MVS
Gaussian Splitting [11]	27.76	0.929	0.084	26.81	0.907	0.140	SfM
PAPR (Ours)	32.07	0.971	0.038	28.72	0.940	0.097	Random
Table 1: Comparison of image quality metrics (PSNR, SSIM and LPIPS [54]) on the NeRF Synthetic dataset [22] and Tanks & Temples subset [13]. Higher PSNR and SSIM scores are better, while lower LPIPS scores are better. All point-based baselines use a total number of 30,000 points and are initialized using the original techniques proposed by their respective authors. Despite being initialized from scratch, our method outperforms all baselines on all metrics for both datasets with the same total number of points.
Figure 5 shows a qualitative comparison between our method, PAPR, and the baselines on the NeRF Synthetic dataset. PAPR produces images with sharper details compared to the baselines. Notably, our method captures fine texture details on the bulldozer's body, the reflections on the drums and the crash, the mesh on the mic, and the reflection on the material ball. In contrast, the baselines either fail to capture the texture, exhibit blurriness in the generated output, or introduce high-frequency noise. These results validate the effectiveness of PAPR in capturing fine details and generating realistic renderings compared to the baselines. For additional qualitative results, please refer to the appendix.
Effect of Number of Points We analyze the impact of the number of points on the rendered image quality in our method. The evaluation is performed using the LPIPS metric on the NeRF synthetic dataset. As shown in Figure 6, our method achieves better image quality by using a higher number of points. Additionally, we compare the performance of our model to that of DPBRF and Point-NeRF. The results show that our model maintains higher rendering quality even with a reduced number of
e
Figure 5: Qualitative comparison of novel view synthesis on the NeRF Synthetic dataset [22]. All point-based methods use a total number of 30,000 points. NPLF [24] fails to capture the texture detail, NeRF [22], SNP [56] and Gaussian Splatting [11] exhibit blurriness in the rendered images, while DPBRF [53] and Point-NeRF [48] display high-frequency artifacts. In contrast, PAPR achieves high-quality rendering without introducing high-frequency noise, demonstrating its ability to capture fine texture details.
points, as few as 1,000 points, compared to the baselines. These findings highlight the effectiveness of our method in representing the scene with high quality while using a parsimonious set of points.
Effect of Ray-dependent Point Embedding Design We analyze our ray-dependent point embedding design by gradually removing components, including the point position  $p_i$  and the displacement vector  $t_{i,j}$ , which are described in Sec. 3.3. Figure 7 shows both the qualitative and quantitative results of this analysis. Removing  $p_i$  in key features introduces increased noise in the learnt point cloud, indicating its importance. Likewise, removing  $t_{i,j}$  in both key and value features results in a failure to learn the correct geometry. These findings validate the effectiveness of our point embedding design and highlight the necessity of both the point position and displacement vector for achieving accurate and high-quality results.
4.4 Practical Applications
Zero-shot Geometry Editing We showcase the zero-shot geometry editing capability of our method by manipulating the positions of the point representations only. In Figure 8a, we present three cases that deform the object geometry: (1) rigid bending of the ficus branch, (2) rotation of the statue's head, and (3) stretching the tip of the microphone to perform a non-volume preserving transformation. The results demonstrate that our model effectively preserves surface continuity after geometry editing, without introducing holes or degrading the rendering quality. Additional results and comparisons with point-based baselines can be found in the appendix.
Object Manipulation In Figure 8b, we present object duplication and removal scenarios. Specifically, we add an additional hot dog to the plate and perform removal operations on some of the balls in the materials scene while duplicating others. These examples demonstrate the versatility of our method in manipulating and editing the objects in the scene.
Texture Transfer We showcase the capability of our method to manipulate the scene texture in Figure 8c. In this example, we demonstrate texture transfer by transferring the associated feature vectors from points corresponding to the mustard to some of the points corresponding to the ketchup. The model successfully transfers the texture of the mustard onto the ketchup, resulting in a realistic and coherent texture transformation. This demonstrates the ability of our method to manipulate and modify the scene texture in a controlled manner.
Exposure Control We showcase the ability of our method to adjust the exposure of the rendered image. To accomplish this, we introduce an additional random latent code input to our feature
renderer  $f_{\theta \mathcal{R}}$  and finetune the model using the cIMLE technique [16]. Figure 8d shows different exposures in the rendered image by varying the latent code input. For more detailed information on this application, please refer to the appendix. This demonstrates our method's flexibility in controlling exposure and achieving desired visual effects in the rendered images.
Figure 6: Ablation study on the effect of the number of points on rendered image quality. The results show that increasing the number of points improves the performance of our method. Moreover, our approach consistently outperforms the baselines when using an equal number of points, while also exhibiting greater robustness to a reduction in the number of points.
Figure 7: Ablation study on the design for the ray-dependent point embedding. We incrementally remove key components in the point embedding, starting with the point position  $p_i$  and then the displacement vector  $t_{i,j}$ . The results show a significant degradation in the learnt geometry and rendering quality at each step of removal. This validates the importance of each component in the design of our point embedding.
**Limitation** Our pruning strategy currently assumes the background image to be given, making it suitable for scenarios with a near-constant background colour. However, this assumption may not hold in more diverse and complex backgrounds. To address this limitation, we plan to explore learning a separate model dedicated to handling background variations in future work.
Figure 8: Applications of PAPR: (a) Zero-shot Geometry editing - deforming objects (bending branch, rotating head, stretching mic), (b) Object manipulation - duplicating and removing objects, (c) Texture transfer - transferring texture from mustard to ketchup, (d) Exposure control - changing from dark to bright.
Societal Impact Expanding our method's capacity with more points and deeper networks can improve rendering quality. However, it's important to consider the environmental impact of increased computational resources, potentially leading to higher greenhouse gas emissions.
Conclusion In this paper, we present a novel point-based scene representation and rendering method. Our approach overcomes challenges in learning point cloud from scratch and effectively captures correct scene geometry and accurate texture details with a parsimoinous set of points. Furthermore, we demonstrate the practical applications of our method through four compelling use cases.
Acknowledgements This research was enabled in part by support provided by NSERC, the BC DRI Group and the Digital Research Alliance of Canada.
[51] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5732-5741, 2021.
[52] Yu-Jie Yuan, Yang tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: Geometry editing of neural radiance fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18332-18343, 2022.
[53] Qian Zhang, Seung-Hwan Baek, Szymon Rusinkiewicz, and Felix Heide. Differentiable point-based radiance fields for efficient view synthesis. SIGGRAPH Asia 2022 Conference Papers, 2022.
[54] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[55] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ArXiv, abs/1805.09817, 2018.
[56] Yiming Zuo and Jia Deng. View synthesis with sculpted neural points. ArXiv, abs/2205.05869, 2022.
[57] Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, and Markus H. Gross. Surface splatting. Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 2001.
A.1 Proximity Attention
As introduced in Sec. 3.3, our model utilizes an attention mechanism to determine the proximity between the ray  $\mathbf{r}_j$  and a point  $i$ . To achieve this, we use three independent embedding MLPs to compute the key  $\mathbf{k}_{i,j}$ , value  $\mathbf{v}_{i,j}$  and query  $\mathbf{q}_j$  for the attention mechanism. The architecture details
of these three embedding MLPs are shown in Figure 9.
Figure 9: Embedding MLPs architecture details.
Figure 10: U-Net details.
A.2 Point Feature Renderer
As described in Sec. 3.3, our point feature renderer uses a modified U-Net architecture to generate colour outputs from the aggregated feature map. The architecture details are shown in Figure 10.
A.3 Point Pruning
As described in Sec. 3.4, we use a background token  $b$  to determine whether a ray intersects with the background. In our experiments, we set  $b = 5$ . Moreover, we modify the calculation of the attention weights  $w_{i,j}$  for each point in Eqn. 6 to incorporate both the background token and the influence score, the updated definition is as follows:
$$
w _ {i, j} = \frac {w _ {i , j} ^ {\prime}}{\sum_ {m = 1} ^ {K} w _ {m , j} ^ {\prime}}, \text {w h e r e} w _ {i, j} ^ {\prime} = \frac {\exp \left(a _ {i , j} \cdot \tau_ {i}\right)}{\exp (b) + \sum_ {m = 1} ^ {K} \exp \left(a _ {m , j} \cdot \tau_ {m}\right)} \tag {10}
$$
Here,  $a_{i,j}$  is defined in Eqn. 6.
Figure 11 shows a comparison between the point cloud learnt by the model with and without pruning on the Lego scene. The results demonstrate that the adopted pruning strategy yields a cleaner point cloud with fewer outliers.
A.4 Point Growing
As mentioned in Sec. 3.4, our method involves growing points in the sparser regions of the point cloud. To identify these sparse regions, we compute the standard deviation  $\sigma_{i}$  of the distances between
Figure 11: Comparison between point clouds learnt from the model without pruning (top row) and the model with pruning (bottom row). The point cloud learnt from the model with pruning exhibits cleaner structure and fewer outliers compared to the point cloud trained without pruning.
Figure 12: Visualization of the clustered point feature vectors. In the bottom row for each scene, each colour represents a feature cluster. The clusters are obtained by clustering the learnt point feature vectors by the K-Means clustering algorithm with  $K = 6$ .
each point  $i$  and its top 10 nearest neighbours. Subsequently, we insert new points near the points with higher values of  $\sigma_{i}$ . The location of the new point near point  $i$  is selected as a random convex combination of  $i$  and its three nearest neighbours.
A.5 Clustering Feature Vectors
To demonstrate the effectiveness of our learnt point feature vectors, we visualize the feature vector clustering in Fig.12. The clustering is obtained by applying the K-Means clustering algorithm with  $K = 6$  to the learnt feature vectors of the points. The resulting clustering reveals distinct separation, indicating the successful encoding of local information at surfaces with different textures and geometries.
More specifically, the feature vectors capture various colours within the scene. In the caterpillar scene, for instance, the point features associated with gray-coloured areas, such as the wheels and the side of the shovel, are clustered together. Moreover, the feature vectors can effectively differentiate between different materials, even when their colours may appear similar. For instance, in the materials scene, two balls on the bottom right most side have similar colours but different finishes—one with a matte appearance and the other with a polished look—and their feature vectors accurately distinguish between them. Additionally, the feature vectors demonstrate the capability to discern
variations in local geometry. In the drum scene, for example, the feature vectors for the large drum in the foreground differ from those representing the smaller drums in the background. These results highlight the encoding of distinct local scene information by the feature vectors, presenting potential utility in applications such as part segmentation.
A.6 Zero-shot Geometry Editing and Object Manipulation
As mentioned in Sec.4.4, our model is capable of zero-shot geometry editing and object manipulation by modifying the scene point cloud. For zero-shot geometry editing, we achieve this by altering the positions of the points, while for object addition or removal, we duplicate or delete specific points accordingly. It is important to note that our approach eliminates the need for any additional pre- or post-processing steps for rendering the scene after the editing process. This shows the simplicity and robustness of our approach in editing the scenes.
A.7 Texture Transfer
As mentioned in Sec. 4.4, to transfer textures between different parts of a scene, we first select the points representing the source and target parts, along with their corresponding point feature vectors. To capture the essential texture information, we identify the top  $K$  principal components for both the source and target feature vectors. To facilitate texture transfer, we project the source feature vectors onto the target principal components and get the corresponding coordinates. Finally, we apply an inverse transformation to bring these coordinates back to the original space along the source principal components. To render an image with transferred textures during testing, we simply replace the texture vectors of the source points with the transferred texture vectors. The remaining components of the model remain unchanged during testing.
Figure 13: Modified point feature renderer architecture for exposure control, where we add an additional latent code input  $\mathbf{z}$  which is fed into an MLP to produce affine transformation parameters for the feature map. The transformed feature map serves as the input to the U-Net architecture to produce the final RGB output image.
A.8 Exposure Control
As exposure level is an uncontrolled variable in novel view synthesis of the scene, there can be multiple possible exposure levels for rendering unseen views not present in the training set. Therefore, we introduce an additional latent variable  $\mathbf{z}$  to the model to explain this variability. Specifically, we modify the architecture of the point feature renderer  $f_{\theta_{\mathcal{R}}}$  and use the latent code  $\mathbf{z} \in \mathbb{R}^{128}$  as an additional input. The latent code is first passed through an MLP, which produces a scaling vector  $\gamma \in \mathbb{R}^{32}$  and a translation vector  $\beta \in \mathbb{R}^{32}$  for the feature map. The scaling vector  $\gamma$  and the translation vector  $\beta$  are applied channel-wise to the feature map. The output is then fed into the U-Net architecture (described in Sec. A.2) to produce the final RGB image. The architecture details are shown in Figure 13.
To effectively train a model to solve this one-to-many prediction problem, we employ a technique known as cIMLE (conditional Implicit Maximum Likelihood Estimation) [16]. cIMLE is specifically useful for addressing the issue of mode collapse and effectively capturing diverse modes within the target distribution. Algorithm 1 shows the pseudo code of cIMLE: