PAPR: Proximity Attention Point Rendering
Authors: Yanshu Zhang
Shichong Peng
Seyed Alireza Moazenipourasil
Ke Li
Venue: NeurIPS 2023 spotlight (2023)
Keywords: point cloud learning
point cloud rendering
TL;DR: A point-based 3D representation and rendering method that can learn scene surface geometry using point cloud from scratch

Abstract: Learning accurate and parsimonious point cloud representations of scene surfaces from scratch remains a challenge in 3D representation learning.  Existing point-based methods often suffer from the vanishing gradient problem or require a large number of points to accurately model scene geometry and texture. To address these limitations, we propose Proximity Attention Point Rendering (PAPR), a novel method that consists of a point-based scene representation and a differentiable renderer. Our scene representation uses a point cloud where each point is characterized by its spatial position, influence score, and view-independent feature vector. The renderer selects the relevant points for each ray and produces accurate colours using their associated features. PAPR effectively learns point cloud positions to represent the correct scene geometry, even when the initialization drastically differs from the target geometry. Notably, our method captures fine texture details while using only a parsimonious set of points. We also demonstrate four practical applications of our method: zero-shot geometry editing, object manipulation, texture transfer, and exposure control. More results and code are available on our project website at https://zvict.github.io/papr/.

PAPR: Proximity Attention Point Rendering
Yanshu Zhang", Shichong Peng', Alireza Moazeni, Ke Li
APEX Lab
School of Computing Science
Simon Fraser University
Lyanshu_zharg shichonngpeeittee seyed alireza moazenipourasily keli pesii
Abstract
Learning accurate and parsimonious point cloud representations of scene surfaces from scratch remains challenge in 3D representation learning. Existing point- based methods often suffer from the vanishing gradient problem require a large number points to accurately model scene geometry texture. To address these limitations, we propose Proximity Attention Point Rendering (P), novel method that consists of a point-based scene representation and a differentiable ren- derer. Our scene representation by its spatial position, influence score, and view-independent feature vector. The renderer selects the relevant points for each ray and produces accurate colours using their associated features. PAPR effectively learns point cloud positions to represent the correct scene geometry, even when the initialization drastically differs from the target geometry. Notably, our method captures fine texture details while using only parsimonious of points. We also demonstrate four practical applications of our method: zero-shot geometry editing, object manipulation, texture transfer, and exposure control. More results and code available our project website!
a point cloud where each point is characterized
1 Introduction
Learning 3D representations is crucial for computer vision and graphics applications. Recent neural rendering methods [22] 42
leverage deep neural networks within the rendering pipelines capture complex geometry and texture details from multi-view RGB images. These methods have many practical applications, such as generating high-quality 3D models [18
50
23
15 48
43 3
and enabling interactive virtual reality experiences [29] 25 (2). However, balancing representation capacity and computational complexity remains a challenge. Large-scale representations offer better quality but demand more resources. In contrast, parsimonious representations strike balance between efficiency and quality, enabling efficient learning, processing, and manipulation of 3D data.
3D representations can be categorized into volumetric and surface representations, Recent advances have shown impressive results in rendering quality, However, the cubic growth encoded information as the scene radius increases makes volumetric representations computationally expensive to process. For example, to render volumetric repre- sentation, all the information along a ray needs to be aggregated, which requires evaluating many samples along each ray. In contrast, surface representations are more parsimonious and efficientt since the encoded information grows quadratically. So surface representations can rendered with much fewer samples.
in volumetric representations [22 48 50
37
17
Efficient surface representations include meshes and surface point clouds. Meshes are difficult to learn from scratch, since many constraints (e.g., no self-intersectton) need to be enforced. Additionally, the topology of the initial mesh fixed, making impossible to change during training 43 In contrast,
Denotes equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Point Cloud Position
(b) Depth (c) Point Feature (d) Rendering
Figure 1: Our proposed method, PAPR, jointly learns a point-based scene representation and a differentiable renderer from scratch using only RGB supervision. PAPR effectively learns the point positions (a) that represent the correct surface geometry, as demonstrated by the depth map (b). Notably, PAPR achieves this even when starting from an initialization that substantially deviates from the target geometry. Furthermore, PAPR learns a view-independent feature vector for each point, capturing the local scene content. The clustering of point feature vectors is shown in (c). The renderer selects and combines these feature vectors to generate high-quality colour rendering (d).
point clouds offer more flexibility in modelling shapes with varying topologies. Hence, in this work, we focus on using point clouds to represent scene surfaces.
Learning a scene representation from scratch requires differentiable renderer that can pass large gradients the representation. Designing a differentiable renderer for point clouds is non-trivial each point is infinitesimal and rarely intersects with rays, it is unclear what the ouiput colour at such rays should be. Previous methods often rely on splat-based rasterization techniques, where points are turned into disks or spheres [45] 15 35
(3 11 [66.. These methods use radial basis function (RBF) kernel to calculate the contribution of cach point to each pixel. However, determining the optimal radius for the splats the RBF kernel is challenging 19], and these methods suffer from the issue of vanishing gradient when the ray is far away from the points, limiting their ability to learn the ground truth geometry that drastically differs from the initial geometry. Additionally, the radius of each splat should be small for accurate texture modelling, and this would require a point cloud with a large number of points, which are difficult to process.
To address these limitations, we introduce novel method called Proximity Attention Point Rendering (PAPR). PAPR consists of a point-based scene representation and differentiable renderer. The scene representation constructed using a point cloud, where each point is defined by its spatial position, an influence score indicating its influence the rendering, and fiew-independent feature vector that captures the local scene content. Our renderer learns to directly select points for each ray and combines them to generate the correct colour using their associated features. Remarkably, PAPR can learn accurate geometry and texture details using a parsimonious set of points, even when the initial point cloud substantially differs from the target geometry, shown in Figure [ We show that the proposed ray-dependent point embedding design is crucial for effective learning of point cloud positions from scratch. Furthermore, datasets demonstrate that PAPR outperforms prior point-based methods in terms of image quality when using parsimonious set of points. We also showcase four practical applications of PAPR: zero-shot geometry editing with part rotations and deformations, object duplication and deletion, texture transfer and exposure control. In summary, our contributions are as follows:
experiments on both synthetic and real- world
1. We propose PAPR, novel point-based scene representation and rendering method that can learn point-based surface geometry from scratch.
2. We demonstrate PAPR's ability capture correct scene geometry and accurate texture details using a parsimonious set of points, leading to an efficient and effective representation.
3. We explore and demonstrate four practical applications with PAPR: zero-shot geometry editing, object manipulation, texture transfer and exposure control.
2 Related Work
Our work focuses on learning and rendering 3D representation, and is most related to differentiable rendering, neural rendering and point-based rendering. While there are many relevant works in these fields, we only discuss the most pertinent ones in this context and refer readers to recent for more comprchensive overview.
surveys 99 41 47

---

Differentiable Rendering For 3D Representation Learning Differentiable rendering plays crucial role in learning 3D representations from data allows gradients to be backpropagated from the rendered output to the representation. Early mesh learning approach, OpenDR [19] approximate gradients using localized Taylor expansion and differentiable filters, but do not fully leverage loss function gradients. Neural Mesh Renderer [10] proposes non-local approximated gradients that utilize gradients backpropagated from a loss function. Some methods [3
modify the forward rasterization step to make the pipeline differentiable by softening object boundaries or making the contribution of triangles to each pixel probabilistic. Several point cloud learning methods,
18 27
such differentiable surface splatting 44] and differentiable Poisson solvers 26), explore surface
reconstruction techniques but they do not focus on rendering fine textures. In contrast, our method jointly learns detailed colour appearance and geometry representations from scratch.
Neural Scene Representation In recent years, there has been growing interest in utilizing neural networks to generate realistic novel views of scenes. Some approaches utilize explicit representations like meshes 42 49), multi-plane images 20) 21 36 55). and point clouds i 30 [5]. Another approach represents the scene as a differentiable density field, as demonstrated by Neural Radiance Field (NcRF) [22]. However, NeRF suffers from slow sampling speed due to volumetric ray marching. Subsequent works have focused on improving training and rendering speed through space discretization [39 6 4
46 and storage optimizations [23 38 50 37 17]. These
methods still rely on volumetric representations, which have a cubic growth in encoded information. In contrast, proposed method leverages surface representations, which more efficient and parsimonious.
Point-based Representation Rendering and Learning Rendering point clouds using ray tracing challenging because they do not inherently define surface, leading an ill-defined intersection between rays and the point cloud. Early approaches used splatting techniques with disks, ellipsoids, surfels [57 32 2 (28). Recent methods 30 [2] [4] incorporate neural networks in the rendering (SfM), Multi- View Stereo (MVS) or LiDAR scanning without optimizing point positions. To learn the point positions, splat- 11 [56) employ radial basis function kernels to compute point contributions pixels but struggle with finding optimal splat kernel radius p and suffer from gradient vanishing when points are far from target pixels. Consequently, these methods require a large number of points to accurately model scene details and are limited to learn small point cloud deformation. In contrast, our method can capture scene details using parsimonious set of points and perform substantial deformation in the initial point cloud to match the correct geometry.
pipeline but assume ground truth point clouds from monectre-frrm--oon
based rasterization methods 45] 15
35
53
Another approach 3 predicts explicit intersection points between rays and optimal meshes recon- structed from point clouds. but it requires auxiliary data with ground truth geometry for learning and RGBD images for point cloud initialization. Our method overcomes these limitations. Xu et al. [48) represent radiance fields with points and use volumetric ray marching. our approach utilizes more efficient and parsimonious surface representations and generates output colour for each ray with only a single forward pass through the network.
3 Proximity Attention Point Rendering
Figure Sguure@provine provides an overview of our proposed end-to-end learnable point- rendering method. The input model includes a set of RGB images along with their corresponding camera intrinsics and extrinsics. Our method jointly learns a point-based scene representation, where each point comprises a spatial position, an influence score, and view-independent feature vector, along with differentiable renderer. This is achieved by minimizing the reconstruction loss between the rendered
image i and the ground truth image Igt according to some distance metric -(, w.r.t the scene representation and the parameters of the differentiable renderer. Unlike previous point cloud learning methods 48 45
B [56] which rely on multi-view stereo (MVS), Structure-frot from- Motion (SfM), depth measurements or object masks for initializing the point cloud positions, our method can learn the point positions from scratch, even when the initialization significantly differs from the target geometry as demonstrated in Figure { In the following sections, we provide more detailed description of each component of our proposed method.
15
35] 53
3

---

Embedding
MLPs
Value features
Feature vector
Key features
Attention
position
1()
Scare Represcatation
G4352)
pesltica
Query Seature
Sanur
directioo
(a) Ray-dependent Point Embedding
(b) Proximity Attention
(c) Point Feature Rendering
Figure 2: An overview of the proposed pipeline for rendering point-based scene representation, where each point is defined by its spatial position, an influence score, and fiew-independent feature vector. (a) Given ray, ray-dependent features are created for each point. These features are used to generate the key, value, and query inputs for a proximity attention model. (b) The attention model selects the points based on their keys and the query, and combines their values to form an aggregated feature. (c) The aggregated feature is gathered for all pixels create feature map. This feature map is then passed through feature renderer, implemented using UNet architecture, to produce the output image. The entire pipeline jointly trained an end-to-end manner using only supervision from the ground truth image.
3.1 Point-based Scene Representation
Our scene representation consists of a set of N points P == {(ps, us, T;) where each point i is characterized by its spatial position u, T that encodes local appearance and geometry and an influence score 7; that represents the influence of the point on the rendering.
R3, an individual view-independent feature vector
Notably, we make the feature vector for each point to be independently learnable, thereby allowing the content encoded in the features to be independent of the distance between neighbouring points. As result, after training, points can be moved different positions without needing make corresponding change in the features. This design also makes it possible to increase the fidelity of the representation by adding more points, thereby capturing more fine-grained details. Additionally, we specifically choose the feature vectors to be hew-independent, ensuring that the encoded information the representation remains invariant to view changes.
Differentiable Point Rendering with Relative Distances
Splat
Par-Point Contrbbtiin
Par-Point Contribtion
Point
Pisel
A
B
C
Image Plane
(a) Hard Boundary Splats
(b) Soft Boundary Splats
(c) Proposed
Comparison of point contributions in splat-based rasterization and the proposed method. the image plane, with the ground truth image shown in the background. The bar charts illustrate the contribution of each point towards specific pixel, represented by the red star. (a) If the points are splats with hard boundaries, none of them intersect with the given pixel, resulting in zero contributions. Consequently, the gradient of the loss at the pixel w.r.t. each point is non-existent, hindering the reduction of loss that pixel. (b) For splats with soft boundaries, although the point contributions are non-zero, they are extremely small, leading to vanishing gradients. (c) Our proposed method normalizes the contribution of points to the given pixel. This ensures that there are always points with substantial contributions, enabling learning point positions from scratch.
Figure 3:
Three points are projected
To learn the scene representation from images from different views, we minimize the reconstruction loss between the rendered output from those views and the ground truth image w.r.t. the scene representation. Typically, we do so with brasee optimization methods, which need to compute

---

the gradient of the loss w.r.t. the scene representation. For this to be possible, the rendered output must be differentiable w.r.t. the representation in other words, the renderer must be differentiable.
In classical rendering pipelines, point clouds are often rendered with splat-based rasterization. However, when attempting to learn point clouds from scratch using this method, several challenges arise for the following reasons. A splat only contributes to the pixels it intersects with, so when a splat does not intersect with specific pixel, it does not contribute to the rendered output that pixel. As result, the gradient of the loss at that pixel w.r.t. the position of the splat is zero. Now, consider a pixel in the ground truth image that no splat intersects with. Even though the loss at that pixel might be high, the gradient w.r.t. the position of every splat is zero, so the loss at that pixel cannot be minimized, as illustrated in Figure Bi.
To address this issue, prior methods have employed a radial basis function (RBF) kernel to soften splat boundaries. This approach makes the contribution of each splat each pixel non-zero. However, even with the soft boundaries, the contribution of a splat to pixel remains small if the pixel is located far away from the centre of the splat. Consequently, for a pixel far away from the centres of all splats, the gradient of the loss at the pixel w.r.t. each splat remains too minuscule to exert a significant influence on its position, as shown in Figure 3p-- - in other words, the gradient vanishes. As result, even though the loss at the pixel might be high, it will take a long time before it is minimized.
We observe that with both hard- and soft-boundan splats, the contribution of each point to given pixel only depends on its own absolute distance to the pixel, without regard to the distances of other points. This gives rise the vanishing gradient problem because the absolute distances for all points can be high. To address this issue, we make the contributions per point towards a pixel depend on relative distances rather than absolute distances. Instead of relying solely on the distance the pixel from a single point to compute its contribution, we normalize the contributions across all points so that the total contribution from all points is always 1. This ensures that there are always points with significant contributions to each pixel, even if the pixel is far away from all points, as shown in Figure 3-
This allows points to be moved far from their initial positions, so the initialization can differ sub- stantially from the ground truth scene geometry. This also means that we can move sufficiently many points from elsewhere to represent the scene geometry faithfully. On the other hand, the learnt representation uses no more points than necessary represent each local part of the scene geometry faithfully, because if there were extraneous points, they could be moved to better represent other parts. The end result is a parsimonious scene representation that uses just enough points to faithfully represent each part of the scene geometry. So, fewer points tend to be allocated to smooth parts of the surface and the interior of opaque surfaces, because having more points there would not improve the rendering quality significantly. Moreover, since the points can be far away from a pixel without having their contributions diminished, we can preserve the surface continuity even if we apply a non-volume preserving transformation (c.g., stretching) to the point cloud post-hoc.
3.3 Implementation of Relative Distances with Proximity Attention
We design an attention mechanism compute contributions from relative distances, which we dub proximity attention. Unlike typical attention mechanisms, the queries correspond to rays and the keys correspond points. To encode the relationship between points and viewing directions, we design an embedding of the points that account for their relative positions to rays. This embedding is then fed in as input to the proximity attention mechanism. We describe the details of each below.
Ray-dependent Point Embedding Given a camera pose C specified by its extrinsics and intrinsics, and sensor resolution H x W, we utilize a pinhole camera model to cast ray ry from the camera centre to each pixel. Each ray r, is characterized by its origin oj E IR3 and unit-length viewing direction d E 3R both defined in the world coordinate system.
To obtain the ray-dependent point embedding of point p; with respect to ray rj, we start by finding the projection p of the point onto the ray:
p o, + (p: - oy, d,) dj
(1)
where '(,,)"' denotes the inner product. Next, we compute the displacement vectors, $tjj and t.y:
$;j p' Oj, ti = PP; - P/:
(2)
5

---

Here, fr captures the depth of the point p; with respect to the camera centre oj along the ray, while t. represents the perpendicular displacement from the ray the point p:. The final ray-dependent point embedding contains these displacement vectors, Si,j and E.j. as well as the point position P:-
Proximity Attention For a given ray rj. start by filtering the top K nearest points p, based on the magnitude their perpendicular displacement vector |t,, This allows us to identify the points that are most relevant for modelling the local contents of the ray.
We adopt the ray- ray-dependent point embedding method described earlier for the closest points, which serves as the input to construct the key in the attention mechanism. For the value branch input, we utilize the K associated point feature vectors u, along with their displacement vectors Sij and L.j. For the input to the query branch, we use the ray direction dj. Additionally, we apply
positional encoding [22 40 r(p) = [sin (2'rp) cos (2' L=6 (== to all components except for the feature vector. These inputs pass through three separate embedding networks S0x- Joy and Sea which are implemented as multi-layer perceptrons (MLPs). The resulting outputs form the final key, value, and query for the attention mechanism:
, fex (h (st.))
(3)
Vi,, for (h7 (Si,) y (tr,)) u:l)
(4)
q fea (d,))
(5)
To compute the aggregated value for the query, we caleulate the weighted sum of the values as f, = -E WayyVe, where each weighting term is calculated by applying the softmax function the raw attention scores az,j. The raw attention score as, is obtained by applying a ReLU function to the dot product between the query qj and the keys k,,, scaled by their dimensionality μ.:
exp(ar.) where oig = max(0, (q k.,))
W.
(6)
Em=1 exp(am.j)
V
By aggregating the features f, for all rays r, corresponding each pixel on the image plane with resolution H x W, we construct the feature map Fe E H xdea for given camera pose C.
Point Feature Renderer To obtain the final rendered colour output image i for given camera pose C, input the aggregated feature map convolutional feature renderer, fox (Fc). Our renderer is based on a modified version of the U-Net architecture 34), featuring two downsampling layers and two upsampling layers. Notably, remove the BatchNorm layers in this network. For more details regarding the model architecture, please refer to the appendix.
3.4 Progressive Refinement of Scene Representation
Point Pruning To eliminate potential outliers during the point position optimization process, we introduce point pruning strategy that utilizes a learnable influence score E R associated with each point. We calculate the probability Wa, E [o, 1] of a ray ry hitting the background by comparing a fixed background token b to the product of each point's 's influence score and its raw attention score as.j (defined Eqn. 6). The probability is given by:
exp
Wb.j
exp(b) + E exp(am.. Tm)
By collecting the background probabilities Wb.j for all x W rays cast from camera pose C, we obtain a background probability mask Pe E #x The rendered output image can then be computed using this background probability mask as follows:
i= (1.- fea (Fe) + Pe Lb
(8)
Here, L, E HW'' represents the given background colour. All influence scores are initialized to zero. Starting from iteration 10, 000, we prune points with < 0 every 500 iterations. Additional details and comparisons to the model without pruning are available in the appendix.

---

Point Growing As mentioned in Sec. 3.1 our scene representation allows for increased modelling capacity by adding more points. To achieve this, we propose point growing strategy that incremen- tally adds points to the sparser regions of the point cloud every 500 iterations until the desired total number of points is reached, More information on how the sparse regions are identified can be found in the appendix. Fig. D illustrates the effectiveness of our point growing strategy in progressively refining the point cloud, resulting a more detailed representation the scene.
3.5 Training Details
Our training objective is minimizing the distance metric d(.--) from the rendered image i to the corresponding ground truth image 1gt. We define the distance metric weighted combination of mean squared error (MSE) and the LPIPS metric [54]. The loss function formulated as:
L = d(i, gt) = MSE(Í, 1 (x) + LPIPPII Ixt)
(9)
We the weight parameters, including u, Ti, 0x, 0v, OQ and dR. We train our model using Adam optimizer [12] on single NVIDIA A100 GPU.
0.01 for all experiments. During training, we jointly optimize model
3.6 Learning Point Positions From Scratch
We demonstrate the effectiveness of our method in learning point positions from scratch using the
Lego scene in the NeRF Synthetic Dataset [22]. We compare our proposed method recent point- based methods, namely DPBRF (53] Point-NeRF 48), NPLF (24) and Gaussian Splatting [1I]. It is worth noting that NPLF does not inherently support point position learning, so we made modifications to their official code to enable point position optimization.
Point-NeRe 48
NPLF 24
DPBRF SE
Gaussian Splatting
PAPR (Ours)
Reference Image
Figure 4: Comparison our method and prior point- point-based methods on learning point cloud positions from scratch. We modify NPLF (24's official implementation All methods start with the same initial point cloud of 1, 0,000 points on a sphere. Point pruning and growing strategies
support point position updates.
disabled for fair comparison of geometry learnability. An RGB image of
the scene from the same view point as the point cloud visualizations is provided as a reference.
Point-NeRF (48) and NPLF (24) fail to reconstruct reasonable point cloud. DPBRF (53) captures rough shape but introduces significant noise and struggles to capture the rear end of the Lego bulldozer effectively. Gaussian Splatting [ captures the Lego silhouette but lacks structural details, such as the bulldozer's 's track (the black belt around the wheels). In contrast, our method successfully learns point cloud that accurately represents the object's surface and captures detailed scene structures.
In our comparison, all methods start with the same set of 1, 000 points initialized on a sphere. To ensure fair evaluation of point cloud position learnability, we disable point pruning and growing techniques for all methods. As shown in Figure 4 our method successfully deforms the initial point cloud to correctly represent the target geometry. contrast, the baselines either fail to recover the geometry, produce noisy results, lack structural details in the learnt geometry.

---

Experiments
To validate our contribution of learning point-based scene representation and rendering pipeline directly from multi-view images, we compare our method to recent point-based neural rendering methods, namely NPLF 24,, DPBRE
53 SNP [56], Point-NeRF 48] and Gaussian Splatting [T]].
As secondary comparison, we demonstrate the potential broader impact of our method by comparing
the widely used implicit volumetric representation method NeRF [22]
To assess the performance of scene modelling with a parsimonious representation, we conduct evaluations using total of 30, 000 points for each method. For all point-based baselines, use their original point cloud initialization methods, which are based on Multi-View Stereo (MVS), Motion (SfM) or visual hull. Conversely, for our approach. we adopt a random point cloud initialization strategy within a predefined volume, introducing an additional level of complexity and challenge to our method. We set the parameter K the point feature vector dimension h = 64. We evaluate all methods in both synthetic and real- world scenarios. For the synthetic setting, we choose the NeRF Synthetic dataset [22), while for the real-world setting, we use the Tanks & Temples [I3] subset, following the same data pre- pre-processing steps as
Structure-froot fro-MMooonno
for selecting the top nearest points, and
48
We evaluate rendered image quality using PSNR, SSIM and LPIPS metrics.
4.1 Quantitative Results
Table I shows the average image quality metric scores. PAPR consistently outperforms the baselines across all metries in both synthetic and real-world settings, without relying on specific initialization. These results demonstrate PAPR's ability to render highly realistic images that accurately capture details using an efficient and parsimonious representation.
NeRF Synthetic
Tank & Temples
PSWR
SSIM f
LPIPS
PSNR t
SSIM
LPIPS +
Initialization
NeRF 122
31.00
0.947
0.081
27.94
0.904
0.168
NPLF 124
18.36
0.780
0.213
21.19
0.761
0.240
MVS
DPBRF 153
25.61
0.884
0.138
17.25
0.634
0.301
Visual Hull
SNP [56]
26.00
0.914
0.110
26.39
0.894
0.160
MVS
NeRF 48)
25.93
0.923
0.129
24.75
0.890
0.184
MVS
Gaussian Splatting 07
27.76
0.929
0.084
26.81
0.907
0.140
SIM
PAPR (Ours)
32.07
0.971
0.038
28.72
0.940
0.097
Random
Table 1: Comparison of image quality metrics (PSNR, SSIM and LPIPS (54)) on the NeRF Synthetic dataset 22 and Tanks & Temples subset [ Higher PSNR and SSIM scores are better, while
lower LPIPS scores are better. All point-based baselines use total number of 30, 000 points and are
initialized using the original techniques proposed by their respective authors. Despite being initialized
from scratch, our method outperforms all baselines on all metrics for both datasets with the same
total number of points.
4.2 Qualitative Results
Figure shows qualitative comparison between our method, PAPR, and the baselines on the NeRF Synthetic dataset. PAPR produces images with sharper details compared to the baselines. Notably, our method captures fine texture details on bulldozer's body, the reflections on the drums and the crash, the mesh on the mic, and the reflection on the material ball. In contrast, the baselines either fail to capture the texture, exhibit blurriness in the generated output, or introduce high- fegh-freeuncy noise. These results validate the effecctiveness of PAPR in capturing fine details and generating realistic renderings compared to the basclines. For additional qualitative results, please refer to the appendix.
4.3 Ablation Study
Effect of Number of Points We analyze the impact of the number of points on the rendered image quality our method. The evaluation performed using the LPIPS metric on the NeRF synthetic dataset. As shown in Figure our method achieves better image quality by using a higher number of points. Additionally, we compare the performance of our model to that of DPBRF and PeRR The results show that our model maintains higher rendering quality even with reduced number of
8

---

Legs
Drums
Mic
Materials
NeRF 22
NPLF [24] DPBRE 533
SNP [66]
Point- NeRF [48]
Gaussian
PAPR (Ours)
Ground Truth
Splat- ting 11
Figure 5: Qualitative comparison of novel view synthesis on the NeRF Synthetic dataset All point-based methods use a total number of 30, 000 points. NPLF (24 fails to capture the texture detail, NeRF [22], SNP (56) and Gaussian Splatting [[]] exhibit blurriness the rendered images, while DPBRF 533 and Point-NeRF (48) display high-frequency artifacts. In contrast, PAPR achieves
high-quality rendering without introducing high-frequency noise, demonstrating ability to capture fine texture details.
points, as few as 1,000 points, compared to the baselines. These findings highlight the effectiveness of our method in representing the scene with high quality while using parsimonious set of points.
Effeet of Ray-dependent Point Embedding Design We analyze our ray-dependent point embed- ding design by gradually removing components, including the point position pi and the displacement vector 4.j. which are described in Sec. 3.3 Figure shows both the qualitative and quantitative results of this analysis. Removing p; key features introduces increased noise the learnt point cloud, indicating its importance. Likewise, removing 4., in both key and value features results in failure learn the correct geometry. These findings validate the effectiveness of our point embedding design and highlight the necessity of both the point position and displacement vector for achieving accurate and high-quality results.
4.4 Practical Applications
Zero-shot Geometry Editing We showcase the zero-shot geometry editing capability of our method by manipulating the positions of the point representations only. Figure (8, we present three cases that deform the object geometry: (1) rigid bending of the ficus branch, (2) rotation of the statue's 's head, and (3) stretching the tip of the microphone to perform non-volume preserving transformation. The results demonstrate that our model effectively preserves surface continuity after geometry editing, without introducing holes or degrading the rendering quality. Additional results and comparisons with point-based baselines can be found in the appendix.
Object Manipulation In Figure Sp, we present object duplication and removal scenarios. Specifi- cally, we add additional hot dog to the plate and perform removal operations some of the balls in the materials scene while duplicating others. These examples demonstrate the versatility of our method in manipulating and editing the objects in the scene.
Texture Transfer We showcase the capability of our method to manipulate the scene texture in Figure (b. In this example, we demonstrate texture transfer by transferring the associated feature vectors from points corresponding to the mustard to some of the points corresponding to the ketchup. The model successfully transfers the texture of the mustard onto the ketchup, resulting in realistic and coherent texture transformation. This demonstrates the ability of our method to manipulate and modify the scene texture in controlled manner.
Exposure Control showease the ability of our method to adjust the exposure of the rendered image. To accomplish this, introduce additional random latent code input to our feature

---

renderer fe and finetune the model using the cIMLE technique [6] Figure 8d shows different exposures the rendered image by varying the latent code imput. For more detailed information on this application, please refer to the appendix. This demonstrates our method's s flexibility in controlling exposure and achieving desired visual effects in the rendered images.
Rendered
Depth
Point
Rendered Image
Depth
Point
Point-NeRF
Image
Cloud
Cloud
w/ p., by w/ ps Full Model
LPIPS- -PP
0.2
0.1
LEES
0.02
10k
20k
30k
Number of points
Figure 6: Ablation study the effect of the number of points on rendered image quality. The results show that increasing the number of points im- proves the performance of method. Moreover, our approach consistently outperforms the baselines when using an equal number of points, while also exhibiting greater robustness to re- duction in the number of points.
Figure 7: Ablation study on the design for the ray-dependent point embedding. We incrementally remove key components in the point embedding. starting with the point position pe and then the displacement vector ttjj The results show
significant degradation in the learnt geometry and rendering
quality at each step of removal. This validates the importance
of each component in the design of our point embedding.
5 Discussion and Conclusion
Limitation Our pruning strategy currently assumes the background image be given, making it suitable for scenarios with a near-constant background colour. However, this assumption may hold more diverse and complex backgrounds. To address this limitation, we plan explore learning separate model dedicated to handling background variations in future work.
Original
After Editing
Deformation
(b) Duplication & Removal
(c) Texture
(d) Exposure Control
Transfer
Figure 8: Applications of PAPR: (a) Zero-shot Geometry editing deforming objects (bending branch, rotating head, stretching mic), (b) Object manipulation duplicating and removing objects, (c) Texture transfer transferring texture from mustard to ketchup, (d) Exposure control changing from dark to bright.
Societal Impact Expanding our method's capacity with more points and deeper networks can improve rendering quality. However, it's important consider the environmental impact of increased computational resources, potentially leading to higher greenhouse gas emissions.
Conclusion In this paper, we present novel point-based scene representation and rendering method. Our approach overcomes challenges in learning point cloud from scratch and effectively captures correct scene geometry and accurate texture details with parsimoinous set of points. Furthermore, we demonstrate the practical applications of our method through four compelling use cases.
Acknowledgements This research was enabled part by support provided by NSERC, the BC DRI Group and the Digital Research Alliance of Canada.
10

---

References
[1] Kara-Ali Aliev, Dmitry Ulyanov, and Victor S. Lempitsky. Neural point- graphics. In European Conference Computer Vision, 2019.
[2] Mario Botsch, Alexander quality surface splatting on today's 's gpus. Proceedings Eurographics/IEE VGTC Symposium Point-Based Graphics, 2005., pages 17-141, 2005.
Hornung, Matthias Zwicker, and Leif P. Kobbelt. High-
[3] Jen-Hao Rick Chang, Wei- Yu Chen, Anurag Ranjan, Kwang Moo Yi, and Oncel Tuzel. Pointer- sect: Neural rendering with cloud-ray intersection. ArXiv, 0tts/2344.12999 2023.
[4] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, 2022.
[5] John Flynn, Michael Broxton, Paul E. Debevec, Matthew DuVall, Graham Fyffe, Ryan S. Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1362-2371,222 2019.
2019.
[6] Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien P. C. Valentin. Fastnerf: High-fidelity neural rendering at 200fps. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14326- 14335, 2021.
[7] Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T. Freeman. Unsupervised training for 3d morphable model regression, 2018 IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 8377-8386, 2018.
[8] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul E. Debevec. Baking neural radiance fields for real-time view synthesis. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5855-5864, 2021.
[9] Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey. ArXiv, abs/2006. 000/2006.105577 12057, 2020.
[10] Hiroharu Kato, Y. Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3907-3916, 2017.
[11] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
[12] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.699 2014.
[13] Arno Knapitsch, Jaesik Park, Qian Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
[14] Georgios Kopanas, Julien Philip. Thomas Leimkühler, and George Drettakis. Point-based neural rendering with vie-view optimization. Computer Graphics Forum, 40, 2021.
[15] Christoph Lassner and Michael Zollhöfer. Pulsar: Efficient sphere-based neural rendering. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1440-1449, 2021.
[16] Ke Li Shichong Peng*, Tianhao Zhang and Jitendra Malik. Multimodal image synthesis with conditional implicit maximum likelihood estimation. International Journal of Computer Vision, May 2020.
[17] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. ArXiv, 0bs/2000.11571, 2020.
[18] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. 2019 IEFF/CVF International Conference on Computer Vision (ICCV), pages 7707-7716, 2019.

---

[19] Matthew Loper and Michael J. Black. Opendr: An approximate differentiable renderer. In European Conference on Computer Vision, 2014.
[20] Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, D. Salesin, William Freeman, and Michael Rubinstein. Layered neural rendering for retiming people in video. ACM Trunsactions on Graphics (TOG), 39:1 - 14, 2020.
[21] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. arXiv: Computer Vision and Pattern Recognition, 2019.
[22] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ArXiv, hhss2003.08934, 2020.
[23] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (TOG), 41:1 15, 2022.
[24] Julian Ost, Issam Hadj Laradji, Alejandro Newell, Yuval Bahat, and Felix Heide. Neural point light fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18398- 11998-18409 18408, 2021.
[25] Keunhong Park, U. Sinha, Jonathan Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Seitz, and Ricardo Martin-Bruun Brualla. Nerfies: Deformable neural radiance fields. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5845-583 2020.
[26] Songyou Peng, Chiyu Max Jiang. Yiyi Liao, Michael Niemeyer, Mare Pollefeys, and Andreas Geiger. Shape points: A differentiable poisson solver. In Neural Information Processing Systems, 2021.
[27] Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or, Pix2vex: Image- to-geometry reconstruction using a smooth differentiable renderer. ArXiv, Abs/1903.11149, 2019.
[28] Hanspeter Pfister, Matthias Zwicker, Jeroen van Baar, and Markus H. Gross, Surfels: surface elements as rendering primitives. Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000.
[29] Albert Pumarola, Enric Corona, Gerard Pons-Moll. and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. 202/ IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages (0313-10322) 2020.
[30] Ruslan Rakhimov, Andrei-Timotei Ardelean, Victor S. Lempitsky, and Evgeny Burnaev. Npbg++: Accelerating neural point-based graphies. ArXiv, abs/2203.13318. 2022.
[31] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14315-14325, 2021.
[32] Liu Ren, Hanspeter Pfister, and Matthias Zwicker. Object space ewa surface splatting: A hardware accelerated approach to high quality point rendering. Computer Graphics Forum, 21, 2002.
[33] Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. A versatile scene model with differentiable visibility applied to generative pose estimation. 2015 IEEE International Conference on Computer Vision (ICCV), pages 765-773, 2015.
[34] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. ArXiv, abs/1505.04597, 2015.
[35] Darius Rückert, Linus Franke, and Mare Stamminger. Adop: Approximate differentiable one-pixel point rendering. ACM Trans. Graph., 41:99:1-99:14, 2021.
12

---

[36] Pratul P. Srinivasan, Richard Tucker. Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 175-184, 2019.
[37] Cheng Sun, Min Sun, and Hwann- Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5449-5459, 2021.
[38] Towaki Takikawa, Alex Evans, Jonathan Tremblay. Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. ACM SIGGRAPH 2022 Conference Proceedings, 2022.
[39] Towaki Takikawa, Joey Litalien, K. Yin, Karsten Kreis, Charles T. Loop, Derek Nowrouuzezhrrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-tine time rendering with implicit 3d shapes. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 11353-11362, 2021.
[40] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. ArXiv, abs/2006.10739, 2020.
[41] Anju Tewari, Otto Fried, Justus Thies, Vincent Sitzmann, S. Lombardi, Z Xu, Tanaba Simon, Matthias Nießner, Edgar Tretschk, L. Liu, Ben Mildenhall, Pranatharthi Srinivasan, R. Pandey, Sergio Esttsasco,,no, S. Fanello, M. Guang Guo, Gordon Wetzstein, y Zhu, Christian Theobalt, Manju Agrawala, Donald B. Goldman, and Michael Zollhöfer. Advances in neural rendering. Computer Graphics Forum, 41, 2021.
[42] Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. arXiv: Computer Vision and Pattern Recognition, 2019.
[43] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Hang Yu, Wei Liu, X. Xue, and Yu- Gang Jiang. Pixel2mesh: 3d mesh model generation via image guided deformation. IEEE Trunsuctions on Puttern Analysis and Machine Intelligence, 43:3600-3613, 2020.
[44] Yifan Wang, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine Sorkine-Horunng. Differen- tiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG), 38:1 - 14, 2019.
[45] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: Snnd-to-end view synthesis from a single image. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2019.
[46] Xiuchao Wu, Jiamin Xu, Zihan Zhu, Hujun Bao, Qixing Huang, James Tompkin, and Weiwei Xu. Scalable neural indoor scene rendering. ACM Transactions on Graphics (TOG), 41:1 16, 2022.
[47] Yiheng Xie. Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 41, 2021.
[48] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Urich Neumann. Point-nerf: Point-based neural radiance fields. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages -428-5438, 2022.
[49] Bangbang Yang, Chong Bao, Junyi Zeng, Hujun Bao, Yinda Zhang, Zhaopeng Cui, and Guofeng Zhang. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. ArXiv, abs/2207. 11911, 2022.
[50] Alex Yu, Sara Fridovich- Keil, Matheew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. 2022 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5491-5500, 2021.
13

---

[51] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa, Plenoctrees for real-time rendering of neural radiance fields. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5732-5741, 2021.
[52] Yu-Jie Yuan, Yang tian Sun, Yu-Kun Lai, Yuewen Ma. Rongfei Jia, and Lin Gao. Nerf-editing: Geometry editing neural radiance fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18332-18343, 2022.
[53] Qian Zhang, Seung- Seung-Hwan Hwan Back, Szymon Rusinkiewicz, and Felix Heide. Differentiable point- based radiance fields for efficient view synthesis. SIGGRAPII Asia 2022 Conference Papers, 2022.
[54] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 386-595,10 2018.
[55] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. ArXiv, abs/1805, 058 17, 2018.
[56] Yiming Zuo and Jia Deng. View synthesis with sculpted neural points. ArXiv, abs/2205.05869, 2022.
[57] Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, and Markus H. Gross. Surface splatting. Proceedings of the 28th annual conference Computer graphics and interactive techniquex, 2001.
14

---

A Implementation Details
A.1 Proximity Attention
As introduced in Sec. [3.3 our model utilizes an attention mechanism to determine the proximity between the ray r and point To achieve this, we use three independent embedding MLPs to compute the key ki.j. value Vi.i and query q for the attention mechanism. The architecture details of these three embedding MLPs are shown Figure
Conv: 12-128, kernel s
Max Pooling
Cony: 128-225, kemel sizc=3
4
E2.
P: P
ReLUU
Positional Encoding: L-6
Pesitional Encodingg: L-66
Positional Encodingg I--6
Pooling
f
query
fex
fey
Paase hd
Conv: 256--512, komel
ReLU
Layer Norm
Layer Narn
Linear: d, -256
Transposed Conv: 512-256, kernel size-1, stride-2
Lincar: -26
Linear: dy -25
ReLU
ReLU
ReLU
Lincar: 256-2256
x4
512-256, kemel
Lincar: 256-x256
Linear: 256-2556
RelUU
ReLU
Layer Norm
Layer Narm
Lincar: 256-332
Transposed Conv: 256- 128, kernel size- stride-2
256-2556
Linear: 256-226
Query y;
Key k..
Value Pa.
Conv: 256--128, kemel
Figure 9: Embedding MLPs architecture details.
ReLU
Linear: 128-33
Figure 10: U-Net details.
A.2
Point Feature Renderer
As described in Sec. our point feature renderer uses a modified U-Net architecture to generate colour outputs from the aggregated feature map. The architecture details are shown in Figure I0
A.3 Point Pruning
As described in Sec. 3.4 we use a background token b to determine whether a ray intersects with the background. In our experiments, we set b 5. Moreover, modify the calculation of the attention weights Way for each point in Eqn. to incorporate both the background token and the influence score, the updated definition is as follows:
w/,
exp(a:.) T.)
Wi.
K -m=
where wh exp(b) Em exp(ama. Tm)
(10)
umm
Here, 0z,j is defined in Eqn.
6
Figure II shows a comparison between the point cloud learnt by the model with and without pruning on the Lego scene. The results demonstrate that the adopted pruning strategy yields a cleaner point cloud with fewer outliers.
A.4 Point Growing
As mentioned in Sec.(3.4 our method involves growing points in the sparser regions the point cloud. To identify these sparse regions, compute the standard deviation σ; of the distances between
15

---

w/o pruning
w pruning
Figure 11: Comparison between point clouds leart from the model without pruning (top row) and the model with pruning (bottom row). The point cloud learnt from the model with pruning exhibits cleaner structure and fewer outliers compared to the point cloud trained without pruning.
Reference
Feature Clusters
Caterpillar
Materials
Drums
Figure 12: Visualization of the clustered point feature vectors. In the bottom row for each scene, each colour represents feature cluster. The clusters are obtained by clustering the learnt point feature vectors by the K-Means clustering algorithm with K
each point and its top 10 nearest neighbours. Subsequently, insert new points near the points with higher values of a; The location of the new point near point i selected as a random convex combination of and its three nearest neighbours.
A.5 Clustering Feature Vectors
To demonstrate the effectiveness of our learnt point feature vectors, we visualize the feature vector clustering in Fig 12 The clustering is obtained by applying the K-Means clustering algorithm with K = to the learnt feature vectors of the points. The resulting clustering reveals distinct separation, indicating the successful encoding of local information at surfaces with different textures and geometries.
More specifically, the feature vectors capture various colours within the scene. In the caterpillar scene, for instance, the point features associated with gray-coloured areas, such as the wheels and the side of the shovel, are clustered together. Moreover, the feature vectors can effectively differentiate between different materials, even when their colours may appear similar. For instance, in the materials scene, two balls on the bottom right most side have similar colours but different finishes- one with a matte appearance and the other with a polished look-and their feature vectors accurately distinguish between them. Additionally, the feature vectors demonstrate the capability to discern
16

---

variations in local geometry. In the drum scene, for example, the feature vectors for the large drum in the foreground differ from those representing the smaller drums the background. These results highlight the encoding of distinct local scene information by the feature vectors, presenting potential utility in applications such as part segmentation.
A.6 Zero-shot Geometry Editing and Object Manipulation
As mentioned in Sec|4.4 our model capable of zero-shot geometry editing and object manipulation by modifying the scene point cloud. For zero-shot geometry editing, we achieve this by altering the positions of the points, while for object addition or removal, we duplicate or delete specific points accordingly. is important to note that our approach eliminates the need for any additional pre- or post-processing steps for rendering the scene after the editing process. This shows the simplicity and robustness of our approach in editing the scenes.
A.7 Texture Transfer
As mentioned Sec. 4.4 to transfer textures between different parts a scene, we first select the points representing the source and target parts, along with their corresponding point feature vectors. To capture the essential texture information, we identify the top K principal components source and target feature vectors. To facilitate texture transfer, we project the source feature vectors onto the target principal components and get the corresponding coordinates. Finally, we apply an inverse transformation to bring these coordinates back to the original space along the source principal components. To render an image with transferred textures during testing, we simply replace the texture vectors of the source points with the transferred texture vectors. The remaining components of the model remain unchanged during testing.
both the
Feature Map
Linear: 128-256
ReLU
Linear: 256-2556
66
ReLU
Lineur: 256-+64
Max/O,
Fenturee Map
UNet
Figure 13: Modified point feature renderer architecture for exposure control, where we add an additional latent code input z which fed into an MLP produce affine transformation parameters for the feature map. The transformed feature map serves as the input to the U-Net architecture to produce the final RGB output image.
A.8 Exposure Control
As exposure level is an uncontrolled variable in novel view synthesis of the scene, there can be multiple possible exposure levels for rendering unseen views not present in the training set. Therefore, we introduce an additional latent variable to the model to explain this variability. Specifically, we modify the architecture of the point feature renderer fèg and use the latent code z R 128 as an additional input. The latent code is first passed through an MLP, which produces scaling vector 7 E 32 and a translation vector R-42 for the feature map. The scaling vector yy and the translation vector / are applied channel-wise to the feature map. The output is then fed into the U-Net architecture (described Sec. A.2 to produce the final RGB image. The architecture details are shown in Figure I3
17

---

To effectively train model to solve this one-to one-to-many prediction problem, we employ a technique known as cIMLE (conditional Implicit Maximum Likelihood Estimation) [[6]. cIMLE is specifically useful for addressing the issue of mode collapse and effectively capturing diverse modes within the target distribution. Algorithm| 1 shows the pseudo code cIMLE:
Algorithm 1 Conditional IMLE Training Procedure
Require:
The set of inputs {x:] ) and the set of corresponding observed outputs {y:l== Initialize the parameters 0 of the generator To for p to Nouter do
Pick random batch S C (1....,m)
for i do
Randomly generate i.idd latent codes
21 Zm
Yi, To(x, 2,) Vj [m] ( + arg min Vis.) Vj E [m]
end for
for g = 1 to N Winerr do
Pick random mini-batch 3 S
0 + V (Eies (1.())))))) //5|
end for end for return 0
In our specific context, x; is the feature map, Tg is the modified point feature renderer, y; is the target RGB image. We first load pre-trained weights for all model parameters except the additional MLP for the latent variable before training. During test time, we randomly sample latent codes to change the exposure of the rendered image.
B Additional Results
B.1 Qualitative Comparison
We include extra qualitative results for NeRF Synthetic dataset in Figure [4] Furthermore, we include qualitative comparisons between our method and the more competitive haselines, based on metric scores, on the Tanks Temples subset in Figure 13
Chair
Ficus
Hotdog
Ship
NeRF 22
NPLF 24
DPBRF [53]
SNP [56]
Point- NeRF 48
Gaussian Splat- ting 11
PAPR (Ours)
Ground Truth
Figure 14: Qualitative comparison of novel view synthesis on the NeRF Synthetic dataset 22
18

---

Barn
Caterpillar
Family
Ignatius
Truck
NeRF 22
SNP 56
Gaussian
PAPR (Ours)
Ground Truth
Splatting
Figure 15: Qualitative comparison of novel view synthesis between method, PAPR, and the more competitive baselines on Tanks Temples 13 subset.
B.2 Point Cloud Learning
We show the point clouds generated by our method for each scene, depicted Figure [6 for the NeRF Synthetic dataset (22). and Figure 17 for the Tanks & Temples [33 subset. These results demonstrate the effectiveness of our method in learning high-quality point clouds that correctly capture the intricate surfaces of the scenes.
Point Cloud Reference
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Figure 16: Point clouds learnt by method the NeRF Synthetic dataset 22
19

---

Reference
m
Point Cloud
Barn
Caterpillar
Family
Ignatius
Truck
Figure 17: Point clouds learnt by our method the Tanks & Temples
13
subset.
B.3 Zero-shot Geometry Editing
present comparisons with point-based baselines on two scenes in which we apply non-volume preserving stretching transformations. In the first scene, we stretch the back of the chair, and in the second case, we stretch the tip the microphone. As shown in Figure [8] NPLF [24), DPBRF [53] Point-NeRF [48] and Gaussian Splatting [1] either create holes produce significant noise after the transformation. Additionally, SNP (56) fails to preserve the texture details following the edits, such as the golden embroidery pattern on the back of the chair and the mesh grid pattern the tip of the microphone, In contrast, our method successfully avoids creating holes and effectively preserves texture details after the transformation.
Before
After
Before
After
NPLF 24!
DPBRF 53
SNP 56
Point-NeRF
Gaussian Splatting [I]
PAPR (Ours)
Figure 18: Qualitative comparisons among method, PAPR, and point-based baselines following non-volume preserving stretching transformations. The rendered images before and after the trans- formations are shown. The results after the transformations are rendered by manipulating the point positions only during test time.
20

---

B.4 Quantitative Results
We provide the metric scores broken down by scene on both datasets. Table shows the per-scene scores for the NeRF Synthetic dataset [22] and Table shows the scores for the Tanks & Temples [3] subset.
Chair
Drums
Lego
Mic
Materials
Ship
Hotdog
Fieus
Avg.
PSNRT
NPLF (24)
19.69
16.45
20.70
19.84
16.03
16.04
18.56
19.55
18.36
DPBRF (53
26.51
19.38
25.09
29. 26
26.20
21.93
31.87
24.61
25.61
SNP [56]
28.81
21.74
25.75
28.17
24.04
23.55
31.74
24.23
26,00
Point-NeRF [88
30.24
23.60
23.42
31.75
24.77
18.72
26.56
28.40
25.93
Gaussian Splatting |
31.40
22.44 44
28.18
31.96
27.39
19.84
36.19
24.66
27.76
PAPRI (urs))
33.59
25.35
32.62
35.64
29.54
26.92
36.40
36.50
32.07
SSIM1
NPLF 24
0.798
0.737
0.763
0.886
0.739
0.678
0.792
0.842
0.780
DPBRF 133
0.929
0.834
0.885
0.951
0.895
0.713
0.953
0.909
0.884
SNP 350
0.938
0.889
0.907
0.964
0.909
0.823
0.961
0.917
0.914
Point-NeRI NeRF [8
0.971
0.927
0.904
0.985
0.929
0.761
0.934
0.973
0.923
Gaussian Splatting
||
0.957
0.901
0.935
0.982
0.938
0.792
0.978
0.953
0 0.929
PAPR (APR(Ours)
0.986
0.951
0.981
0.993
0,972
0.904
0.988
0.994
0.971
LPIPS Vgg
NPLF 14
0.183
0.214
0.236
0.136
0.226
0.339
0.226
0.143
0.213
DPBRF [33
0.098
0.174
0.142
0.070
0.143
0.2888
0.093
0.095
0.138
SNP 56
0.049
0.081
0.057
0.025
0.072
0.167
0.036
0.050
0.1 10
Point-NeRF 48
0.065
0.125
0.152
0.038
0.148
0.298
0.143
0.067
0.129
Gaussian Splattig ||
0.049
0.108
0.077
0.019
0.063
0.273
0.037
0.049
0.084
PAPR(Ours)
0.018
0.055
0.027
0.007
0.036
0.129
0.021
0.010
0.038
Table 2: Comparison of image quality metrics (PSNR, SSIM and LPIPS [54) )) broken down by scene,
for the NeRF Synthetic dataset 22
B.5 Ablation Study
provide additional quantitative results for the ablation study. Figure 19 shows PSNR and SSIM scores for the methods using different number of points. Additionally, T the PSNR and SSIM scores for the various choices of the ray-dependent point embedding. These results validate the effectiveness of our proposed method.
30
0.9
PSNR
SSIM
20
0.8
Ours Point-NeRF DPBRF
Ours Point-NeRF DPBRF
10
0.7
1k
10k
20k
30k
1k
10k
20k
30k
Number of points
Number of points
Figure 19: Extra image quality metrics (PSNR, SSIM) for the ablation study different number of points. Higher values of PSNR and SSIM scores are better.
21

---

Ignatius
Truck
Barn
Caterpillar
Family
Avg.
PSNR
NPLF (24
24.60
20.01
20.24
17.69
23.39
21.19
DPBRF (53
18.22
16.20
14.68
17.23
19.93
17.25
SNP (5
28.41
24.32
25.27
22.62
31.31
26.39
Point-NeRF
48
28.59
25.57
20.97
17.71
30.89
24.75
Gaussian Splatting
26.48
22.32
26.22
23.27
35.79
26.81
PAPR(Ours)
28.40
26.98
27.06
26.79
34.39
28.72
SSIMT
NPLF (24
0.882
0.725
0.634
0.719
0.846
0.761
DPBRF (53
0.797
0.563
0.422
0.614
0.776
0.634
SNP [50]
0.944
0.876
0.832
0.858
0.958
0.894
Point-NeRF 48
0,960
0.926
0.799
0.797
0.970
0.890
Gaussian Splatting [7]
0.937
0.867
0.860
0.889
0.983
0.907
PAPR(Ours)
0.956
0.931
0.896
0.932
0.983
0.940
LPIPS
NPLF (24]
0.120
0.265
0.381
0.275
0.157
0.240
DPBRF 53
0.174
0.317
0.484
0.334
0.194
0.301
SNP [56]
0.077
0.177
0.278
0.196
0.071
0.160
Point-NeRF 48
0.085
0.162
0.355
0.237
0.084
0.184
Gaussian Splatting 117
0.095
0.186
0.231
0.155
0.032
0.140
PAPR(Ours)
0.072
0.108
0.157
0.118
0.031
0.097
Table 3: Comparison of image quality metrics (PSNR, SSIM and LPIPS [54) for the Tanks & Temples [13 subset.
broken down by scene,
Lego
Mic
PSNR
SSIM 1
PSNK 1
SSIM
Full Model
32.62
0.981
35.64
0.993
w/o
31.67
0.976
32.00
0.985
w/o Pt, 14,)
13.26
0.701
18.73
0.902
Table 4: Extra image quality metrics (PSNR, SSIM) for the ablation study on different designs for the ray-dependent point embedding. Higher values of PSNR and SSIM scores better.
22


Reviews:
[{"review_id": "vGMrRcqyiI", "reviewer": "Reviewer_QEnw", "summary": "This paper tackles the problem of point-based rendering and proposes a new point-based differentiable rendering pipeline with proximity attention. With an attention-based point rendering, the proposed pipeline avoids the tuning of points' radius or splatting sizes that typical point rendering requires. Such a design enables efficient and high-quality rendering. The effectiveness is verified on NeRF synthetic and Tank and Temple datasets.", "questions": "See above.", "limitations": "The authors provide discussion of limitations in the paper.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed proximity attention is neat and effective.\n\n1. The paper is well-written and easy to follow.\n\n2. The experiments are solid.", "weaknesses": "### 1. Choices of query, key, value for attention\n\na. I think the paper misses an important ablation of various representations for query, key and value for the proximity attention. Currently, query is represented as ray direction, key is used as $(t, s, \\mathbf{p})$, and value is represented as $(t, s, \\mathbf{u})$. A natural question is what if 1) we remove $s$ in both key and value; 2) we have key and value are both represented as $(t, s, \\mathbf{p}, \\mathbf{u})$ (adding feature vector)?\n\nb. What is related is Fig. 7. Can authors clarify: for \"incrementally remove key components in the point embedding\" (Fig. 7's caption), does it mean removing $t_{i,j}$ in both key and value representation?\n\n### 2. Relationship to [24]\n\na.  I feel like the proximity attention is quite similar/related to the **Ray-centric Point Encoding** in [24]. Essentially, [24] also proposes an attention-based point embedding, where \"query\" is the ray direction same as this paper; \"key\" and \"value\" are concatenations from points's positions, distance between point and ray, angle between points and ray (also similar to the paper).\n\nb. Therefore, to better position the paper, I want to understand the difference between this paper and [24]. This is one of the major motivations why I am curious about the ablations about the choices of query/key/values.\n\nc. One difference I can see is that [24] uses an MLP as the renderer while PAPR uses a UNet. Can the author provide an ablation where an MLP is used for $f_{\\theta_\\mathcal{R}}$ (L181)? If PAPR does not work well with MLP, then it seems like the major improvement may come from the renderer's capabilities instead of the proposed attention-based point rendering.\n\nd. Another difference is that [24] uses an image-based feature as the point's feature. Such a design does not allow for changing points's positions if the points's features are not updated later. For Fig. 4, authors state \"we made modifications to their official code to enable point position optimization\" (L214-215). Can the authors clarify whether the features will be updated or not? If the features are not updated, then this needs clarifications and it may be better to replace [24] results in Fig. 4 with initializing from a sparse point cloud of SfM or MVS. This will not shade the advantage of PAPR.\n\ne. Similarly to d above, do the results of [24] in Tab. 1 come from initializing from a sphere? If so, I think it is better to replace it with initializing from a sparse point cloud and give a note in the table about this.\n\n### 3. Speed\n\na. An advantage of traditional point-based rendering [15, 53] is the speed. Can authors also provide the inference speed of PAPR? \n\nb. What is related is the training speed. Can authors also provide the training time of PAPR?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Additional ablation studies on key and value design for the proximity attention.\n\nA1: Sure. We conduct the ablation study on the key and value design using the NeRF Synthetic dataset. \n\n|  | PSNR | SSIM | LPIPS-vgg |\n| --- | --- | --- | --- |\n| No s in both K and V | 30.11 | 0.963 | 0.049 |\n| Use (s,t,p,u) for K and V | 30.91 | 0.968 | 0.041 |\n| Original K and V | 32.07 | 0.971 | 0.038 |\n\nAs shown in the results above, our proposed combination achieves the best performance.\n\n### Q2: How were the different point encoding components removed in the ablation study  (Fig. 7)?\n\nA2: We first remove p from the key, then we remove t in **both** key and value. \n\n### Q3: Relationship to [24]\n\nA3: Indeed, our method is different from [24] in various ways, including point embedding design, renderer architecture and point feature. \n\n**Point Embedding Design:** Our point embedding is simpler than [24], and our experimental results validate the effectiveness of our method in learning the scene geometry and rendering high quality results. \n\n**Renderer Architecture:** We conduct the ablation study where we replace the UNet architecture for $f_{\\theta_{\\mathcal{R}}}$ with an MLP on the NeRF Synthetic dataset. The results are shown in the table below.\n\n| Renderer Architecture | PSNR | SSIM | LPIPS-vgg |\n| --- | --- | --- | --- |\n| MLP | 30.54 | 0.962 | 0.054 |\n| U-Net (Original) | 32.07 | 0.971 | 0.038 |\n\nAs shown above, our model with the modified renderer architecture still significantly outperforms [24] in terms of all metrics. This validates the efficacy of our method does not solely come from the renderer’s architecture. Nevertheless, we do find that the UNet architecture provide a bit of improvement to the rendering quality in our case.\n\n**Point Feature:** As the reviewer pointed out, another distinction is [24] used image-based features, whereas our point features are view-independent and do not depend on images. Yes, for [24], we updated the features jointly with the point coordinates, including for the results in Fig. 4. Despite this modification, [24] failed to capture the scene geometry correctly.  \n\n**Whether [24] was initialized with the original initialization method:** Actually, for all the results in Tab. 1, we ran the baseline methods with their original point initialization methods. In contrast, our method was initialized from a random point cloud. So we gave all baselines an advantage over our method, which makes the evaluation setting more challenging. In particular, the point positions of [24] are initialized using a sparse point cloud obtained from COLMAP. \n\nWhile it was not feasible to study and evaluate the effect of every facet of our design due to resource limitations, we have included our code with this submission to facilitate a thorough exploration of the finer details by reviewers.\n\n### Q4: Can authors also provide the inference speed of PAPR?\n\nA4: Certainly. To render an 800x800 image, our model takes 2.8s, while NeRF would take 11.8s. Our code's inference speed is not yet optimized, with the standard PyTorch topk function constituting about 96% of total inference runtime. Utilizing custom CUDA kernels could potentially accelerate inference speed further.\n\n### Q5: Can authors also provide the training time of PAPR?\n\nA5: Of course. It takes about 20 hours to train a model for a scene from either NeRF Synthetic or Tanks & Temples datasets. In comparison, NeRF would require 20 hours to train on a scene from the NeRF synthetic dataset, and 30 hours on a scene from Tanks & Temples subset."}, {"review_id": "v4EtswAHQv", "reviewer": "Reviewer_fb1M", "summary": "This paper presents a novel point-based rendering approach, PAPR, to learn a point cloud for neural rendering under the supervision of posed RGB images. At the core of softening the gradient flow by using an attention mechanism in a novel and elegant way, the proposed PAPR enables high-fidelity rendering while explicitly learning the parsimonious point-based scene representation from scratch. Based on this, the proposed PAPR enjoys flexibility for geometry editing, object manipulation, texture transfer, and exposure control, all with impressive results. ", "questions": "Because this paper has shown their ability to learn a parsimonious point cloud from scratch, what will be if we have some inductive initialization of the point cloud? For example, could the proposed method refine the sparse reconstruction quality if we have a sparse reconstruction result for the scene by COLMAP? I am also curious about the potential of doing bundle adjustment with the proposed PAPR.\n", "limitations": "The authors have discussed their limitations. I don't have any concern on this point.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Learning a parsimonious point cloud from scratch in the form of neural rendering without using the 3D scene information is very interesting and has many potential applications. \n\n2. The core module, proximity attention, is well-designed and should be computationally efficient. Instead of sampling points along rays, the proximity attention module proposed in this paper uses the projection relationship between rays and points to aggregate the proximate points in an attention form and feed the point features in a U-Net for rendering. The overall pipeline is neat and technically sound. The computation should be efficient as well.\n", "weaknesses": "1. Because the authors emphasize the parsimony property of the point cloud, the number of points should be the key to be studied. Although the authors have provided an ablation study in Fig. 6 on the effect of the number of points on rendered image quality, some in-depth discussion would further strengthen this paper. Because the authors mentioned that the proposed method is currently suitable for the near-constant background color, I am curious about the parsimony of the point cloud concerning the non-constant background color. \n\n2. The growing scheme is somewhat heuristic with predefined hyperparameters such as 500 iterations, top-10 nearest neighbors, etc. There should be some ablation study for the effect of these hyperparameters. Furthermore, because proximity attention could capture the relationship between points, is there a way to use the learned affinity to identify the sparse regions?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The parsimony of the point cloud concerning the non-constant background color.\n\nA1: The parsimony of the point cloud is not related to the presence of a non-constant background colour, as we use the points to model the foreground scenes exclusively. To handle non-constant background, one idea is to learn a dedicated background model (e.g., a light field-based or NeRF-based model). The key is to separate the modelling of the foreground from the background so that the parsimony of the foreground representation is preserved.\n\n### Q2: Ablation on the effects of point growing hyperparameters.\n\nA2: We show the results of different point growing interval  on the NeRF Synthetic dataset in the table below.\n\n| Point Growing Interval (# of iterations) | PSNR $\\uparrow$ | SSIM $\\uparrow$ | $LPIPS_{vgg} \\downarrow$ |\n| --- | --- | --- | --- |\n| 100 | $31.39$ | $0.969$ | $0.041$ |\n| 2000 | $31.35$ | $0.968$ | $0.037$ |\n| 500 (Original) | $32.07$ | $0.971$ | $0.038$ |\n\nAs shown, our model performance is not very sensitive to the point growing interval.\n\nWe also show the results of using different numbers of top nearest neighbours (NN) to determine the sparse region in the table below.\n\n| Number of NNs | PSNR $\\uparrow$ | SSIM $\\uparrow$ | $LPIPS_{vgg} \\downarrow$ |\n| --- | --- | --- | --- |\n| 5 | $31.54$ | $0.969$ | $0.041$ |\n| 20 | $31.58$ | $0.970$ | $0.040$ |\n| 10 (Original) | $32.07$ | $0.971$ | $0.038$ |\n\nAs shown, our model performance is not very sensitive to the numbers of top nearest neighbours (NN) for determining the sparse region either.\n\n### Q3: Is there a way to use the learned affinity to identify the sparse regions?\n\nA3: That is an interesting question. One idea is to use the entropy of the attention weights as a heuristic for identifying the regions for point growing. Specifically, if the attention weights have large entropy, it could mean that the model is less certain about which points to combine to produce the final colour. Therefore, more points may be helpful in that region.    \n\n### Q4: Could the proposed method refine the sparse reconstruction quality if we have a sparse reconstruction result for the scene by COLMAP? I am also curious about the potential of doing bundle adjustment with the proposed PAPR.\n\nA4: Yes, these are definitely possible and will be exciting avenues to explore in future works."}, {"review_id": "6JVn77q8vu", "reviewer": "Reviewer_dttq", "summary": "This paper proposes a novel method that consists of a point-based scene representation and a differentiable renderer. The proposed representation utilizes relative distance to characterize the contribution of each point, preventing the vanishing gradient issue. The method outperforms baselines in terms of optimizing point cloud positions, rendering. And it enables interesting applications.", "questions": "- How can the transparent material modeled with the proximity attention?\n\n- How this approach can be extended to complicated scene with lot of occlusions?\n\n- This approach can work on rigid deformation for some extent. How can this approach be extended to non-rigidly deforming scene?", "limitations": "- Clarifying some details (as mentioned in the weakness section) could further improve the paper. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overview figure is clear, well-conveying the idea of the paper.\n\n- The representation is simple, reasonable and well-desinged.\n\n- The proposed method outperforms the baselines.", "weaknesses": "The paper is sound in general. However there are several points that can be clarified.\n\n- Feature vector: learned or deterministically determined? If it is the latter case, how it can be determined?\n\n- In the figure 3-(c), why does the gradient scale increases again when the target pixel position deviates much from the point cloud?\n\n- In L148-151, it is mentioned that using the relative distance guarantess that there are always some points contributing to each pixel, even if their absolute distances are large.\nHowever, if the distance is too far, wouldn't this design enlarge the error, i.e., generating the color of the unrelated point cloud? I think setting a threshold would be helpful to prevent error, or is this thresholding same as the filtering process in L164?. Also, if the filtering is applied (L164), wouldn't the points that are far away be removed and therefore cannot contribute to the pixel? And how is this filtering threshold decided? Would the value of it affect the performance?   \n\n- How can the view dependent effect be modeled? From my understanding, different viewing direction results in different s,t value of each point, and thus it is possible to model it?\n\n- What is the fixed background token b in L189?\n\n- In Figure 5, the result of proposed method on the Mic sample shows very fine-grained detail similar to GT. However, the result on the Materials sample is relatively blurry.\nWhat is the reason behind the quality difference?\n\n- It would be great if the visual ablations can be done when using the different number of points.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Is feature vector learned or deterministically determined?\n\nA1: The feature vector is learnt as mentioned in L207 - 208.\n\n### Q2: In the figure 3-(c), why does the gradient scale increases again when the target pixel position deviates much from the point cloud?\n\nA2: As the pixel moves to the right away from i, it first gets close to j and then moves past j. As it approaches j, it gets a lot closer to j than i, and so the ratio of its distance to i to the sum of its distances to i and j gets large, and so the scale of the gradient w.r.t. i gets small. However, as it moves past j and gets far from it, the ratio of its distance to i to the sum of its distances to i and j eventually decreases to 1/2, and so the the scale of the gradient w.r.t. i gets large again.\n\n### Q3: If the distance is too far, wouldn't this design (using relative distance) enlarge the error, i.e., generating the color of the unrelated point cloud?\n\nA3: No. The “error” mentioned here is actually the supervision signal, which is essential for learning from scratch. In cases where the model produces a colour at a distant pixel that has a high loss value, the training process would update the point positions and feature vectors to produce a colour that better matches the ground truth, consequently reducing the error. Therefore, when the training has converged, the model would have learnt to produce the correct colour for every pixel, including the ones that are distant from the points.        \n\n### Q4: I think setting a threshold would be helpful to prevent error, or is this thresholding same as the filtering process in L164?\n\nA4: No, the filtering process in L164 is different from thresholding. To illustrate the difference, imagine a scene where one region contains points lying on a sparse grid, while an adjacent region contains points on a dense grid. Now, let's examine the rays intersecting these two regions.\n\nIf we were to apply a single threshold to rays at both regions, there would be more points that fall within the threshold in the dense region than the sparse region. However, by applying the filtering process in L164, we can ensure that there would be exactly K points selected near each ray, regardless of the region's density. This is unattainable through thresholding.\n\n### Q5: Also, if the filtering is applied (L164), wouldn't the points that are far away be removed and therefore cannot contribute to the pixel? And how is this filtering threshold decided? Would the value of it affect the performance?\n\nA5: No, the filtering in L164 does not necessarily result in the removal of points that are far away. Instead, this process retains the top K nearest points around the ray, irrespective of their absolute distances from the ray. i.e., this approach ensures that even in cases where all points are far away from the pixel, the top K nearest points can still contribute to it.\n\nRegarding the impact of K on performance, using an excessively small K value could restrict the contextual information available to the attention mechanism, potentially leading to worse performance. In practice, we choose value of K=20 that is big enough to provide sufficient context to the model while maintaining a reasonable computational load. \n\n### Q6: How can the view dependent effect be modeled? From my understanding, different viewing direction results in different s,t value of each point, and thus it is possible to model it?\n\nA6: Yes, that’s right.\n\n### Q7: What is the fixed background token b in L189?\n\nA7: The background token enables the attention mechanism to handle background rays by assigning low weights to all points. Without this term “b”, the attention mechanism would not be able to do so. In our experiments, we set b=5, as specified in Section B.3 of the supplementary material.\n\n### Q8: What is the reason behind the quality difference between the mic and the material sample in Figure 5?\n\nA8: The quality difference is because the Mic scene only has a single object with all of the points on its surface, whereas the material scene needs to distribute the same total number of points among twelve balls, resulting in less detail per ball.\n\n### Q9: It would be great if the visual ablations can be done when using the different number of points.\n\nA9: Please refer to the rebuttal PDF for the qualitative results of the ablation on different number of points.  \n\n### Q10: How can the transparent material modeled with the proximity attention?\n\nA10: Fully transparent material can be modelled simply with an absence of points. If you mean translucent material, they may conceivably be modelled with a lower foreground score, which would cause a shift in the attention weights towards other points that are more opaque. \n\n### Q11: How this approach can be extended to complicated scene with lot of occlusions?\n\nA11: This could be achieved by increasing both the total number of points and the hyperparameter K, which controls the filtering for the top K nearest point around each ray. \n\n### Q12: How can this approach be extended to non-rigidly deforming scene?\n\nA12: Yes. In fact, we’ve showcased a non-rigid stretching deformation of the mic in the third column of Figure 8a."}, {"review_id": "J1MkV8knu6", "reviewer": "Reviewer_TJqY", "summary": "This paper presents a novel view synthesis and 3D reconstruction framework via a point-based representation and a jointly learned transformer-based differentiable renderer. Given multi-view images of a foreground-centric scene, the method can effectively deform a point-cloud from scratch to match the surfaces, with an optional pruning and growing mechanism. Each point is associated with a learned feature, which allows novel views of the scene to be rendered through a transformer-based point renderer.\n\nExperiments show that this method leads to high-quality geometry and novel view synthesis results, significantly outperforming existing point-based NVS methods as well as the vanilla NeRF.", "questions": "No critical concerns. It would be helpful to elaborate on the generalization of the jointly learned transformer renderer.", "limitations": "The paper includes a nice discussion on the limitation regarding background modeling.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "### S1 - A solid pipeline for point-based reconstruction and rendering\n- The proposed pipeline is very well designed and validated. In particular:\n    - The design of the transformer-based point renderer is interesting. Ray direction is treated as the query feature, point position the key, and point features the value, which make a lot of sense.\n    - The transformer directly outputs the weights of each point for integration, without a separate volume rendering function as in Point-NeRF that would introduce another order of computation.\n\n### S2 - Great results on geometry reconstruction and novel view synthesis\n- The method effectively deforms a point cloud from a initial sphere to complex surfaces, which is a very compact representation of the scene geometry, and allows for various scene editing.\n- The NVS results are also quite impressive, even outperforming NeRF (although I guess one might be able to further tune NeRF to achieve even better numbers).\n\n### S3 - Good demonstrations of scene editing with points\n- The paper presents a number of scene editing applications, including deformation, object removal and appearance editing. Having an explicit point-based representation makes such editing much easier than implicit NeRF/volumes.\n\n### S4 - Good presentation\n- The paper is well written, with clear motivation and illustrative figures.", "weaknesses": "### W1 - Renderer seems to be learned jointly on one scene only\n- In theory, the transformer-based renderer should be generic and can be shared across any scene?\nDoes it generalize to other scenes?\n\n### (minor) W2 - Technical details\n- There seems no explicit inductive bias in the renderer that would encourage it to actually put higher weights to points closer to the ray and to the camera (within the K nearest points). Does the learned renderer actually utilizes the information of $\\mathbf{s}_{i,j}$ and $\\mathbf{t}$ as we expect? It would be helpful to visualize the weights.\n- Sup. mat. Sec. B.3 Eq (10): Is the $w_{i,j}$ here taken before or after the softmax in Eq. (6)?\n    - If before, $w_{i,j}$ is not guaranteed to be positive, and Eq. (7) and Eq. (10) would be problematic, as both $w_{i,j}$ and $\\tau_i$ can be either positive or negative.\n    - If after, there would be double softmax / exponential. Does that cause any issue, eg. vanishing gradients?\n- Pruning points with $\\tau_i \\lt 0$ seems quite arbitrary. I suppose this is related to the choice of the predefined background weight scalar $b$. Is the threshold $0$ chosen empirically? (why not, say, $\\tau_i \\lt 1$ or $\\tau_i \\lt -2.64$?)\n- What is the run time? How does it compare to say NeRF?\n- What happens if it applied to forward facing scenes with complex background, eg. LLFF? The limitation discusses the failure in modeling background in 360° scenes, but what about forward-facing scenes, where points can be legitimately assigned to \"background\"?\n\n### Additional comments\n- Fig. 4: When the pruning is disabled, it is not surprising at all that Point-NeRF fails to represent the surface using the points, as it is still using a volume rendering function where the underlying geometry is ultimately still represented by the density values. Although, I agree it is great that the proposed method is able to deform the points correctly around the surface without pruning, thanks to the distance-aware weighting in the rendering function I suppose.\n- Only one attention layer?\n- Any points remain inside the object?\n- Line 109: minor grammatical issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: In theory, the transformer-based renderer should be generic and can be shared across any scene? Does it generalize to other scenes?\n\nA1: Agreed, the architecture should generalize and it would be an interesting direction to explore in future work.\n\n### Q2: Does the learned renderer actually utilizes the information of $s_{i, j}$ and $t$ as we expect?\n\nA2: We have included a visualization of the attention weights of the points for a given ray in the rebuttal PDF. As shown in the figure, the attention mechanism does place higher weights on points closer to the ray. \n\n### Q3: Sup. mat. Sec. B.3 Eq (10): Is the $w_{i, j}$ here taken before or after the softmax in Eq. (6)?\n\nA3: Good catch. There was a typographical error in the equation. The $w_{i, j}$ should indeed be before the softmax, and should have had a ReLU applied to ensure non-negativity. Concretely, the softmax in Eq. (6) should have been moved outside of the definition of $w_{i, j}$, and ReLUs should be applied on $w_{i, j}$ in Eqs. (7) and (10). So the correct formulas for Eqs. (6), (7) and (10) are:\n\n$w_{i,j} = \\frac{<\\mathbf{q}_{j}, \\mathbf{k}_{i,j}>}{\\sqrt{\\mu}}$\n\n$w_{b,j} = \\frac{\\text{exp}(b)}{\\text{exp}(b) + \\sum^K_{m=1}\\text{exp}(\\max(0, w_{m,j})\\cdot \\tau_m)}$\n\n$a_{i,j} = \\frac{\\text{exp}(\\max(0, w_{i,j})\\cdot \\tau_i)}{\\text{exp}(b) + \\sum^K_{m=1}\\text{exp}(\\max(0, w_{m,j})\\cdot \\tau_m)}$\n\nWe will update the formulas in the camera-ready. \n\n### Q4: I suppose the pruning threshold is related to the choice of the predefined background weight scalar $b$. Is the threshold for $\\tau_i$ chosen empirically?\n\nA4: Actually the pruning threshold is not related to the choice of $b$. It’s chosen to be 0 based on the behaviour of the attention weights as a function of $\\tau_i$. Specifically, if $\\tau_i$ is negative, then the higher $\\max(0, w_{m,j})$ gets, the lower the attention weight becomes. Intuitively, this means that the point is a distractor and so should be pruned. \n\n### Q5: What is the run time? How does it compare to say NeRF?\n\nA5: To render an 800x800 image, our model takes 2.8s, while NeRF would take 11.8s. Our code's inference speed is not yet optimized, with the standard PyTorch topk function constituting about 96% of total inference runtime. Utilizing custom CUDA kernels could potentially accelerate inference speed further.\n\n### Q6: What happens if it applied to forward facing scenes with complex background, eg. LLFF, where points can be legitimately assigned to \"background”?\n\nA6: This depends on whether the background is given. If not, then the method would would face the same issue as the 360-degree scenes. \n\n### Q7: Only one attention layer?\n\nA7: Yes.\n\n### Q8: Any points remain inside the object?\n\nA8: While there may be some points inside, the vast majority are on the surface.\n\n### Q9: Line 109: minor grammatical issues.\n\nA9: Thank you for pointing it out, we will revise it in the camera-ready."}, {"review_id": "o9ggY9Er0r", "reviewer": "Reviewer_U5yB", "summary": "This paper proposes a neural point renderer that improves the existing differentiable point rasterizers using a learned attention block, which improves the gradient behavior alleviating the \"vanishing gradient\" issue that comes with many existing differentiable point rasterizer. The method compares favorably with prior point-based NeRFs, jointly optimizing the point features and point locations, and achieving impressive 2D visual and 3D reconstruction quality even when using only poorly initialized sparse points. Additionally, it demonstrates useful applications thanks to the explicit point representation.", "questions": "I'd like the authors to address my concerns in the \"weakness\" section. Besides, please also help me understand the following questions in the rebuttal.\n\n1. Since the relative distance is completely learned without any prior, why would the network learn to produce higher weights for closer points. Could you visualize the learned weights for some sampled rays? Please also clarify whether Fig 3c is computed from a real learned result or plotted as the \"desired behavior\"? In addition, why do the curves go up again when the points move further away from the pixel (the right-most part of the blue curve and the left-most part of the orange curve).\n\n1. (cont) The inputs to the key and query embeddings don't share any similarity at all. The MLPs need to learn to map them to compatible space and ensure that keys (3) of points that are close to the rays (which would correspond to the Fig 3c gradient behavior) are more similar to the query (5). How does the network learn this without any prior? Do you need special initialization for the MLPs? And what is the convergence rate.\n\n2. Ideally the learnable foreground score should be related depth and occlusion. But without constraint, this learned foreground score can be abused to label \"outlier points\" that are far from any target pixels due to optimization problems, which can be conveniently pruned and therefore ignored in the optimization. Please visualize the foreground score before point pruning to ensure that this score indeed correspond to visibility.\n\n", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method uses a learned attention block to determine the blending weights. It engineers the query, value and key embeddings using suitable geometric entities, which seems oddly effective (see my questions below) to emulate the similar effect of a fully deterministic point rasterizer in terms of the blending behavior w.r.t. point relative distance and depth.\n- The learning is further assisted with progressive point growing and pruning, which, as demonstrated in the ablation experiments, are effective techniques to achieve high quality reconstruction.\n- The visualization of point features through clustering is very interesting and insightful.\n- The paper is well-written, carefully explained, and the proposed components are thoroughly evaluated.\n- The improvement over existing point-based NeRF papers are significant.", "weaknesses": "1. No discussion about complexity. This is important, since the proposed method boasts to be able to compute gradient for all points regardless of their absolution distance to the target pixel in the pixel-space. This requires computing the attention with *every point* for every ray. Existing methods using RBF or other definition rasterization gradient (e.g. [44]) utilize handcrafted rules to reduce the impact of a pixel to a point as the point-pixel increases. While these rules may have the vanishing gradient problem the authors identified, they can serve as hard constraint to limit the number of target pixels required to evaluate the gradient for a specific point, i.e. pixels outside a search window can be ignored. With the proposed approach, and the claimed benefit, that each point receives gradient from all pixels, the computation is significantly higher. It's therefore important to breakdown the algorithm complexity of each step.\n\n1 (cont). Besides the theoretical complexity analysis, the authors should also provide the training and inference speed compared to a standard NeRF that achieves similar visual quality.\n\n2. the remark to [44] is not correct. [44] *is* able to render colored and shaded points. Besides, SynSin [Wiles et.al. CVPR 2020], Pulsar [Lassner, Christoph, and Michael Zollhofer. CVPR 2021] and Adop [35] are three relevant papers that should be cited too. In addition, I would argue that [44], Pulsar and [35] are more relevant than the baselines that are based on MVS initialization, and should be compared too.\n\n3. The necessity to use a convolutional U-Net in the end is disappointing, as the convolution is likely to induce flickering and other inconsistencies when moving the camera, which could be the cause of the flickering on drum surface in the video?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Computation complexity of attention per ray.\n\nA1: Actually we don't compute the attention with every point for each ray. As mentioned on L164-166, instead of attending to all points, we attend to the top K nearest points around each ray. This “K” serves as a hard constraint for the computation cost for each ray. In our experiments, we choose K=20 which is a lot smaller than the overall point count of 30k.\n\n### Q2: Training and inference speed?\n\nA2: To render an 800x800 image, our model takes 2.8s, while NeRF would take 11.8s. Our code's inference speed is not yet optimized, with the standard PyTorch topk function constituting about 96% of total inference runtime. Utilizing custom CUDA kernels could potentially accelerate inference speed further.\n\nIn terms of training time, our method takes about 20 hours to train a model for a scene from either NeRF Synthetic or Tanks & Temples datasets. In comparison, NeRF would require 20 hours to train on a scene from the NeRF synthetic dataset, and 30 hours on a scene from Tanks & Temples subset.\n\n### Q3: Remark on [44].\n\nA3: Thank you for the correction - we will revise in the camera-ready.\n\n### Q4: SynSin [Wiles et al.], Pulsar [Lassner et al.] and Adop [35] should be cited too.\n\nA4: In fact, these works have been cited already, as [45], [15] and [35].\n\n### Q5: Comparison to [44], [15] and [35].\n\nA5: Given the limited timeframe of the rebuttal period, we managed to run two of the more recent baselines, Pulsar [15] and Adop [35], and include the results of the comparison in the rebuttal PDF. As shown, our method outperforms the baselines and contains more details in the rendering.\n\nThe DSS [44] codebase is older and so it is no longer compatible with later versions of CUDA that are required by our GPU hardware. Despite investing substantial time and effort, we did not manage to upgrade their codebase to be compatible with our current hardware setup.  \n\n### Q6: The necessity to use a convolutional U-Net in the end is disappointing, as the convolution is likely to induce flickering and other inconsistencies when moving the camera.\n\nA6: Sure, the use of convolutional U-Net may cause flickering. However, it's important to note that using a convolutional U-Net isn't mandatory for the renderer. In fact, we present an ablation study below, wherein the U-Net is replaced by an MLP on the NeRF Synthetic dataset.\n\n| | PSNR | SSIM | LPIPS-vgg |\n| --- | --- | --- | --- |\n| MLP | 30.54 | 0.962 | 0.054 |\n| U-Net  | 32.07 | 0.971 | 0.038 |\n\nThe table above demonstrates that even with the substitution of an MLP for the renderer, our model maintains competitive performance. Importantly, our primary contribution doesn't hinge on the specific choices of renderer architecture.\n\n### Q7: Since the relative distance is completely learned without any prior, why would the network learn to produce higher weights for closer points. Could you visualize the learned weights for some sampled rays?\n\nA7: Certainly. The visualization of the attention weights of the points around a given ray is shown in the rebuttal PDF. As shown, the model does indeed assign higher weights to closer points. Intuitively, if the attention mechanism assigns high weights for points that are far away, it would be difficult to preserve the 3D consistency. For instance, when two rays with different camera centers both shoot rays towards the same point in the foreground scene, the rendered colour should be very similar. However, when the attention favours points far away, there could be many far away points for each ray to choose from, and it’s easy to choose different points. This would result in rendering a different colour for each different view for the same 3D location. Therefore, the best way to preserve consistency in rendered colour is by assigning high attention weight to points close to the ray.\n\n### Q8: Please also clarify whether Fig 3c is computed from a real learned result or plotted as the \"desired behavior\"?\n\nA8: Fig 3c is the “desired behaviour”.\n\n### Q9: Why do the curves go up again when the points move further away from the pixel?\n\nA9: As the pixel moves to the right away from i, it first gets close to j and then moves past j. As it approaches j, it gets a lot closer to j than i, and so the ratio of its distance to i to the sum of its distances to i and j gets large, and so the scale of the gradient w.r.t. i gets small. However, as it moves past j and gets far from it, the ratio of its distance to i to the sum of its distances to i and j eventually decreases to 1/2, and so the the scale of the gradient w.r.t. i gets large again.\n\n### Q10: How does the network learn to map the key and query to similar space without any prior?\n\nA10: Similar to what is explained earlier, the network is encouraged to assign high weights to closer points to better preserve 3D consistency. Therefore, the MLPs would learn a feature space where the proximity between the point embedding and the ray corresponds to the proximity between their respective features, thereby allowing closer points to receive higher weights.\n\n### Q11: Do you need special initialization for the MLPs?\n\nA11: No. The weights to the MLPs are randomly initialized.\n\n### Q12: What is the convergence rate?\n\nA12: We have include a loss curve throughout the course of training in the rebuttal PDF. As shown, our model converges smoothly and quickly.\n\n### Q13: Please visualize the foreground score.\n\nA13: Actually the learnable foreground score is not necessarily related to depth and occlusion, because we assume a constant coloured background, which does not need to be modelled with points. Hence, all points are visible from some views. Instead it is used to upweight the contributions of particular points, and so is more related to the certainty in the model’s belief in the point being in the foreground and its opaqueness. We include a visualization of the foreground scores before pruning in the rebuttal PDF. As shown, they behave as intended. "}]

Author rebuttal:
# General Responses

We thank all reviewers for dedicating their time, providing insightful feedback, and unanimously acknowledging the efficacy of the proposed method and the good results. In particular, the reviewers remarked, "the proposed proximity attention is neat and effective”(R QEnw), "the overall pipeline is neat and technically sound." (R fb1M), "the representation is simple, reasonable and well-designed." (R dttq), “Solid pipeline, great results, good demonstrations and presentation” (R TJqY), “The visualization is very interesting and insightful, the proposed components are thoroughly evaluated and the improvement are significant” (R U5yB).

We found the comments and questions very helpful, which add value to our work. Here, we provide a one-sentence summary of our response to selected questions raised by the reviewers — please refer to our individual responses to each review for the details.

### Q1: Please visualize the attention weights. Does the learned renderer actually utilizes the information of $s_{i, j}$ and $t$ as we expect? (R ****TJqY, U5yB)****

A1: Yes, the attention weight visualization in the rebuttal PDF does indeed show higher weights for closer points.

### Q2: Is feature vector learned or deterministically determined? (R ****dttq)****

A2: The feature vector is learnt as mentioned in L207 - 208.

### Q3: How can this approach be extended to non-rigidly deforming scene? (R ****dttq)****

A3: Yes. In fact, we’ve showcased a non-rigid stretching deformation of the mic in the third column of Figure 8a. 

### Q4: How were the different point encoding components removed in the ablation study  (Fig. 7)? (R ****QEnw)****

A4: We first remove p from the key, then we remove t in **both** key and value. 

### Q5: Only one attention layer? (R ****TJqY)****

A5: Yes.

### Q6: Computational complexity of attention per ray. (R ****U5yB)****

A6: The complexity is O(K), where K is the k parameter in the top-k operation as described on L164-166.

### Q7: Do you need special initialization for the MLPs? (R ****U5yB)****

A7: No. The weights to the MLPs are randomly initialized.

Meta-review: A lot of support for this work. ``U5yB`` was still slightly negative, but their concerns are not perceived as blocking by the AC. AC tends to understand what the paper proposes and is supportive, too.

A small observation not related to or affecting this decision: Authors were quite pushy, asking reviewers and AC to respond. Please could you consider abstaining from this next time? It just creates a lot of noise in an already stressful process, thanks!
Decision: Accept (spotlight)