Title: PAPR: Proximity Attention Point Rendering
Venue: NeurIPS 2023
Accepted: yes
Keywords: point cloud learning

point cloud rendering
Abstract: Learning accurate and parsimonious point cloud representations of scene surfaces from scratch remains a challenge in 3D representation learning.  Existing point-based methods often suffer from the vanishing gradient problem or require a large number of points to accurately model scene geometry and texture. To address these limitations, we propose Proximity Attention Point Rendering (PAPR), a novel method that consists of a point-based scene representation and a differentiable renderer. Our scene representation uses a point cloud where each point is characterized by its spatial position, influence score, and view-independent feature vector. The renderer selects the relevant points for each ray and produces accurate colours using their associated features. PAPR effectively learns point cloud positions to represent the correct scene geometry, even when the initialization drastically differs from the target geometry. Notably, our method captures fine texture details while using only a parsimonious set of points. We also demonstrate four practical applications of our method: zero-shot geometry editing, object manipulation, texture transfer, and exposure control. More results and code are available on our project website at https://zvict.github.io/papr/.

PAPR: Proximity Attention Point Rendering

Abstract

Learning accurate and parsimonious point cloud representations of scene surfaces from scratch remains a challenge in 3D representation learning. Existing pointbased methods often suffer from the vanishing gradient problem or require a large number of points to accurately model scene geometry and texture. To address these limitations, we propose Proximity Attention Point Rendering (PAPR), a novel method that consists of a point-based scene representation and a differentiable renderer. Our scene representation uses a point cloud where each point is characterized by its spatial position, influence score, and view-independent feature vector. The renderer selects the relevant points for each ray and produces accurate colours using their associated features. PAPR effectively learns point cloud positions to represent the correct scene geometry, even when the initialization drastically differs from the target geometry. Notably, our method captures fine texture details while using only a parsimonious set of points. We also demonstrate four practical applications of our method: zero-shot geometry editing, object manipulation, texture transfer, and exposure control. More results and code are available on our project website.

1 Introduction

Learning 3D representations is crucial for computer vision and graphics applications. Recent neural rendering methods [22, 42, 50, 23, 15, 48] leverage deep neural networks within the rendering pipelines to capture complex geometry and texture details from multi-view RGB images. These methods have many practical applications, such as generating high-quality 3D models [18, 43, 3, 7] and enabling interactive virtual reality experiences [29, 25, 52]. However, balancing representation capacity and computational complexity remains a challenge. Large-scale representations offer better quality but demand more resources. In contrast, parsimonious representations strike a balance between efficiency and quality, enabling efficient learning, processing, and manipulation of 3D data.

3D representations can be categorized into volumetric and surface representations. Recent advances in volumetric representations [22, 48, 50, 37, 17] have shown impressive results in rendering quality. However, the cubic growth in encoded information as the scene radius increases makes volumetric representations computationally expensive to process. For example, to render a volumetric representation, all the information along a ray needs to be aggregated, which requires evaluating many samples along each ray. In contrast, surface representations are more parsimonious and efficient since the encoded information grows quadratically. So surface representations can be rendered with much fewer samples.

Efficient surface representations include meshes and surface point clouds. Meshes are difficult to learn from scratch, since many constraints (e.g., no self-intersection) need to be enforced. Additionally, the topology of the initial mesh is fixed, making it impossible to change during training [43]. In contrast,

(a) Point Cloud Position

(b) Depth

(c) Point Feature (d) Rendering

point clouds offer more flexibility in modelling shapes with varying topologies. Hence, in this work, we focus on using point clouds to represent scene surfaces.

Learning a scene representation from scratch requires a differentiable renderer that can pass large gradients to the representation. Designing a differentiable renderer for point clouds is non-trivial each point is infinitesimal and rarely intersects with rays, it is unclear what the output colour at such rays should be. Previous methods often rely on splat-based rasterization techniques, where points are turned into disks or spheres [45, 15, 35, 53, 11, 56]. These methods use a radial basis function (RBF) kernel to calculate the contribution of each point to each pixel. However, determining the optimal radius for the splats or the RBF kernel is challenging [9], and these methods suffer from the issue of vanishing gradient when the ray is far away from the points, limiting their ability to learn the ground truth geometry that drastically differs from the initial geometry. Additionally, the radius of each splat should be small for accurate texture modelling, and this would require a point cloud with a large number of points, which are difficult to process.

To address these limitations, we introduce a novel method called Proximity Attention Point Rendering (PAPR). PAPR consists of a point-based scene representation and a differentiable renderer. The scene representation is constructed using a point cloud, where each point is defined by its spatial position, an influence score indicating its influence on the rendering, and a view-independent feature vector that captures the local scene content. Our renderer learns to directly select points for each ray and combines them to generate the correct colour using their associated features. Remarkably, PAPR can learn accurate geometry and texture details using a parsimonious set of points, even when the initial point cloud substantially differs from the target geometry, as shown in Figure 1. We show that the proposed ray-dependent point embedding design is crucial for effective learning of point cloud positions from scratch. Furthermore, our experiments on both synthetic and real-world datasets demonstrate that PAPR outperforms prior point-based methods in terms of image quality when using a parsimonious set of points. We also showcase four practical applications of PAPR: zero-shot geometry editing with part rotations and deformations, object duplication and deletion, texture transfer and exposure control. In summary, our contributions are as follows:

We propose PAPR, a novel point-based scene representation and rendering method that can learn point-based surface geometry from scratch.

We demonstrate PAPR's ability to capture correct scene geometry and accurate texture details using a parsimonious set of points, leading to an efficient and effective representation.

We explore and demonstrate four practical applications with PAPR: zero-shot geometry editing, object manipulation, texture transfer and exposure control.

2 Related Work

Our work focuses on learning and rendering 3D representation, and it is most related to differentiable rendering, neural rendering and point-based rendering. While there are many relevant works in these fields, we only discuss the most pertinent ones in this context and refer readers to recent surveys [9, 41, 47] for a more comprehensive overview.

Differentiable Rendering For 3D Representation Learning Differentiable rendering plays a crucial role in learning 3D representations from data as it allows gradients to be backpropagated from the rendered output to the representation. Early mesh learning approach, OpenDR [19], approximate gradients using localized Taylor expansion and differentiable filters, but do not fully leverage loss function gradients. Neural Mesh Renderer [10] proposes non-local approximated gradients that utilize gradients backpropagated from a loss function. Some methods [33, 18, 27] modify the forward rasterization step to make the pipeline differentiable by softening object boundaries or making the contribution of triangles to each pixel probabilistic. Several point cloud learning methods, such as differentiable surface splatting [44] and differentiable Poisson solvers [26], explore surface reconstruction techniques but they do not focus on rendering fine textures. In contrast, our method jointly learns detailed colour appearance and geometry representations from scratch.

Neural Scene Representation In recent years, there has been a growing interest in utilizing neural networks to generate realistic novel views of scenes. Some approaches utilize explicit representations like meshes [42, 49], multi-plane images [5, 20, 21, 36, 55], and point clouds [1, 30, 15]. Another approach represents the scene as a differentiable density field, as demonstrated by Neural Radiance Field (NeRF) [22]. However, NeRF suffers from slow sampling speed due to volumetric ray marching. Subsequent works have focused on improving training and rendering speed through space discretization [39, 6, 8, 4, 31, 51, 46] and storage optimizations [23, 38, 50, 37, 17]. These methods still rely on volumetric representations, which have a cubic growth in encoded information. In contrast, our proposed method leverages surface representations, which are more efficient and parsimonious.

Point-based Representation Rendering and Learning Rendering point clouds using ray tracing is challenging because they do not inherently define a surface, leading to an ill-defined intersection between rays and the point cloud. Early approaches used splatting techniques with disks, ellipsoids, or surfels [57, 32, 2, 28]. Recent methods [1, 30, 24, 14] incorporate neural networks in the rendering pipeline but assume ground truth point clouds from Structure-from-Motion (SfM), Multi-View Stereo (MVS) or LiDAR scanning without optimizing point positions. To learn the point positions, splatbased rasterization methods [45, 15, 35, 53, 11, 56] employ radial basis function kernels to compute point contributions to pixels but struggle with finding optimal splat or kernel radius [9] and suffer from gradient vanishing when points are far from target pixels. Consequently, these methods require a large number of points to accurately model scene details and are limited to learn small point cloud deformation. In contrast, our method can capture scene details using a parsimonious set of points and perform substantial deformation in the initial point cloud to match the correct geometry.

Another approach [3] predicts explicit intersection points between rays and optimal meshes reconstructed from point clouds, but it requires auxiliary data with ground truth geometry for learning and RGBD images for point cloud initialization. Our method overcomes these limitations. Xu et al. [48] represent radiance fields with points and use volumetric ray marching, our approach utilizes more efficient and parsimonious surface representations and generates output colour for each ray with only a single forward pass through the network.

3 Proximity Attention Point Rendering

Figure 2 provides an overview of our proposed end-to-end learnable point-based rendering method. The input to our model includes a set of RGB images along with their corresponding camera intrinsics and extrinsics. Our method jointly learns a point-based scene representation, where each point comprises a spatial position, an influence score, and a view-independent feature vector, along with a differentiable renderer. This is achieved by minimizing the reconstruction loss between the rendered image ˆ I and the ground truth image I gt according to some distance metric d ( · , · ) , w.r.t. the scene representation and the parameters of the differentiable renderer. Unlike previous point cloud learning methods [48, 45, 15, 35, 53, 11, 3, 56] which rely on multi-view stereo (MVS), Structure-fromMotion (SfM), depth measurements or object masks for initializing the point cloud positions, our method can learn the point positions from scratch, even when the initialization significantly differs from the target geometry as demonstrated in Figure 4. In the following sections, we provide a more detailed description of each component of our proposed method.

(a) Ray-dependent Point Embedding

(b) Proximity Attention

(c) Point Feature Rendering

Embedding

MLPs

UNet

Valuefeatures

t

S

featurevector

S

Keyfeatures

Attention

t

S

position

(1)p = T

PointScene Representation

(x,y,z)

position

Queryfeature

ERh

featurevector

ray direction

ER

influence score

3.1 Point-based Scene Representation

Our scene representation consists of a set of N points P = { ( p i , u i , τ i ) } N i =1 where each point i is characterized by its spatial position p i ∈ R 3 , an individual view-independent feature vector u i ∈ R h that encodes local appearance and geometry and an influence score τ i ∈ R that represents the influence of the point on the rendering.

Notably, we make the feature vector for each point to be independently learnable, thereby allowing the content encoded in the features to be independent of the distance between neighbouring points. As a result, after training, points can be moved to different positions without needing to make a corresponding change in the features. This design also makes it possible to increase the fidelity of the representation by adding more points, thereby capturing more fine-grained details. Additionally, we specifically choose the feature vectors to be view-independent, ensuring that the encoded information in the representation remains invariant to view changes.

3.2 Differentiable Point Rendering with Relative Distances

Splat

Per-Point Contribution

Per-Point Contribution

Per-Point Contribution

C

B

A

A

B

C

A

B

C

A

B

C

Image Plane

(a) Hard Boundary Splats

(b)Soft Boundary Splats

(c) Proposed

To learn the scene representation from images from different views, we minimize the reconstruction loss between the rendered output from those views and the ground truth image w.r.t. the scene representation. Typically, we do so with gradient-based optimization methods, which need to compute

the gradient of the loss w.r.t. the scene representation. For this to be possible, the rendered output must be differentiable w.r.t. the representation - in other words, the renderer must be differentiable.

In classical rendering pipelines, point clouds are often rendered with splat-based rasterization. However, when attempting to learn point clouds from scratch using this method, several challenges arise for the following reasons. A splat only contributes to the pixels it intersects with, so when a splat does not intersect with a specific pixel, it does not contribute to the rendered output at that pixel. As a result, the gradient of the loss at that pixel w.r.t. the position of the splat is zero. Now, consider a pixel in the ground truth image that no splat intersects with. Even though the loss at that pixel might be high, the gradient w.r.t. the position of every splat is zero, so the loss at that pixel cannot be minimized, as illustrated in Figure 3a.

To address this issue, prior methods have employed a radial basis function (RBF) kernel to soften splat boundaries. This approach makes the contribution of each splat to each pixel non-zero. However, even with the soft boundaries, the contribution of a splat to a pixel remains small if the pixel is located far away from the centre of the splat. Consequently, for a pixel far away from the centres of all splats, the gradient of the loss at the pixel w.r.t. each splat remains too minuscule to exert a significant influence on its position, as shown in Figure 3b - in other words, the gradient vanishes. As a result, even though the loss at the pixel might be high, it will take a long time before it is minimized.

We observe that with both hard- and soft-boundary splats, the contribution of each point to a given pixel only depends on its own absolute distance to the pixel, without regard to the distances of other points. This gives rise to the vanishing gradient problem because the absolute distances for all points can be high. To address this issue, we make the contributions per point towards a pixel depend on relative distances rather than absolute distances. Instead of relying solely on the distance to the pixel from a single point to compute its contribution, we normalize the contributions across all points so that the total contribution from all points is always 1. This ensures that there are always points with significant contributions to each pixel, even if the pixel is far away from all points, as shown in Figure 3c.

This allows points to be moved far from their initial positions, so the initialization can differ substantially from the ground truth scene geometry. This also means that we can move sufficiently many points from elsewhere to represent the scene geometry faithfully. On the other hand, the learnt representation uses no more points than necessary to represent each local part of the scene geometry faithfully, because if there were extraneous points, they could be moved to better represent other parts. The end result is a parsimonious scene representation that uses just enough points to faithfully represent each part of the scene geometry. So, fewer points tend to be allocated to smooth parts of the surface and the interior of opaque surfaces, because having more points there would not improve the rendering quality significantly. Moreover, since the points can be far away from a pixel without having their contributions diminished, we can preserve the surface continuity even if we apply a non-volume preserving transformation (e.g., stretching) to the point cloud post-hoc.

3.3 Implementation of Relative Distances with Proximity Attention

We design an attention mechanism to compute contributions from relative distances, which we dub proximity attention . Unlike typical attention mechanisms, the queries correspond to rays and the keys correspond to points. To encode the relationship between points and viewing directions, we design an embedding of the points that account for their relative positions to rays. This embedding is then fed in as input to the proximity attention mechanism. We describe the details of each below.

Ray-dependent Point Embedding Given a camera pose C specified by its extrinsics and intrinsics, and a sensor resolution H × W , we utilize a pinhole camera model to cast a ray r j from the camera centre to each pixel. Each ray r j is characterized by its origin o j ∈ R 3 and unit-length viewing direction d j ∈ R 3 , both defined in the world coordinate system.

To obtain the ray-dependent point embedding of point p i with respect to ray r j , we start by finding the projection p ′ i of the point onto the ray:

where ' ⟨· , ·⟩ ' denotes the inner product. Next, we compute the displacement vectors, s i,j and t i,j :

Here, s i,j captures the depth of the point p i with respect to the camera centre o j along the ray, while t i,j represents the perpendicular displacement from the ray to the point p i . The final ray-dependent point embedding contains these displacement vectors, s i,j and t i,j , as well as the point position p i .

Proximity Attention For a given ray r j , we start by filtering the top K nearest points p i based on the magnitude of their perpendicular displacement vector || t i,j || . This allows us to identify the points that are most relevant for modelling the local contents of the ray.

We adopt the ray-dependent point embedding method described earlier for the K closest points, which serves as the input to construct the key in the attention mechanism. For the value branch input, we utilize the K associated point feature vectors u i , along with their displacement vectors s i,j and t i,j . For the input to the query branch, we use the ray direction d j . Additionally, we apply positional encoding [22, 40] γ ( p ) = [ sin ( 2 l πp ) , cos ( 2 l πp )] L =6 l =0 to all components except for the feature vector. These inputs pass through three separate embedding networks f θ K , f θ V and f θ Q , which are implemented as multi-layer perceptrons (MLPs). The resulting outputs form the final key, value, and query for the attention mechanism:

To compute the aggregated value for the query, we calculate the weighted sum of the values as f j = ∑ K i =1 w i,j v i,j , where each weighting term is calculated by applying the softmax function to the raw attention scores a i,j . The raw attention score a i,j is obtained by applying a ReLU function to the dot product between the query q j and the keys k i,j scaled by their dimensionality µ :

By aggregating the features f j for all rays r j corresponding to each pixel on the image plane with resolution H × W , we construct the feature map F C ∈ R H × W × d feat for a given camera pose C .

Point Feature Renderer To obtain the final rendered colour output image ˆ I for a given camera pose C , we input the aggregated feature map to a convolutional feature renderer, ˆ I = f θ R ( F C ) . Our renderer is based on a modified version of the U-Net architecture [34], featuring two downsampling layers and two upsampling layers. Notably, we remove the BatchNorm layers in this network. For more details regarding the model architecture, please refer to the appendix.

3.4 Progressive Refinement of Scene Representation

Point Pruning To eliminate potential outliers during the point position optimization process, we introduce a point pruning strategy that utilizes a learnable influence score τ i ∈ R associated with each point. We calculate the probability w b,j ∈ [0 , 1] of a ray r j hitting the background by comparing a fixed background token b to the product of each point's influence score τ i and its raw attention score a i,j (defined in Eqn. 6). The probability is given by:

By collecting the background probabilities w b,j for all H × W rays cast from camera pose C , we obtain a background probability mask P C ∈ R H × W × 1 . The rendered output image ˆ I can then be computed using this background probability mask as follows:

Here, I b ∈ R H × W × 3 represents the given background colour. All influence scores are initialized to zero. Starting from iteration 10 , 000 , we prune points with τ i < 0 every 500 iterations. Additional details and comparisons to the model without pruning are available in the appendix.

Point Growing As mentioned in Sec. 3.1, our scene representation allows for increased modelling capacity by adding more points. To achieve this, we propose a point growing strategy that incrementally adds points to the sparser regions of the point cloud every 500 iterations until the desired total number of points is reached. More information on how the sparse regions are identified can be found in the appendix. Fig. 1a illustrates the effectiveness of our point growing strategy in progressively refining the point cloud, resulting in a more detailed representation of the scene.

3.5 Training Details

Our training objective is minimizing the distance metric d ( · , · ) from the rendered image ˆ I to the corresponding ground truth image I gt . We define the distance metric as a weighted combination of mean squared error (MSE) and the LPIPS metric [54]. The loss function is formulated as:

We set the weight λ to 0 . 01 for all experiments. During training, we jointly optimize all model parameters, including p i , u i , τ i , θ K , θ V , θ Q and θ R . We train our model using Adam optimizer [12] on a single NVIDIA A100 GPU.

3.6 Learning Point Positions From Scratch

We demonstrate the effectiveness of our method in learning point positions from scratch using the Lego scene in the NeRF Synthetic Dataset [22]. We compare our proposed method to recent pointbased methods, namely DPBRF [53], Point-NeRF [48], NPLF [24] and Gaussian Splatting [11]. It is worth noting that NPLF does not inherently support point position learning, so we made modifications to their official code to enable point position optimization.

Point-NeRF [48]

NPLF [24]

DPBRF [53]

In our comparison, all methods start with the same set of 1 , 000 points initialized on a sphere. To ensure a fair evaluation of point cloud position learnability, we disable point pruning and growing techniques for all methods. As shown in Figure 4, our method successfully deforms the initial point cloud to correctly represent the target geometry. In contrast, the baselines either fail to recover the geometry, produce noisy results, or lack structural details in the learnt geometry.

4 Experiments

To validate our contribution of learning point-based scene representation and rendering pipeline directly from multi-view images, we compare our method to recent point-based neural rendering methods, namely NPLF [24], DPBRF [53], SNP [56], Point-NeRF [48] and Gaussian Splatting [11]. As a secondary comparison, we demonstrate the potential broader impact of our method by comparing it to the widely used implicit volumetric representation method NeRF [22].

To assess the performance of scene modelling with a parsimonious representation, we conduct evaluations using a total of 30 , 000 points for each method. For all point-based baselines, we use their original point cloud initialization methods, which are based on Multi-View Stereo (MVS), Structure-from-Motion (SfM) or visual hull. Conversely, for our approach, we adopt a random point cloud initialization strategy within a predefined volume, introducing an additional level of complexity and challenge to our method. We set the parameter K = 20 for selecting the top nearest points, and the point feature vector dimension h = 64 . We evaluate all methods in both synthetic and real-world scenarios. For the synthetic setting, we choose the NeRF Synthetic dataset [22], while for the real-world setting, we use the Tanks & Temples [13] subset, following the same data pre-processing steps as in [48]. We evaluate rendered image quality using PSNR, SSIM and LPIPS [54] metrics.

4.1 Quantitative Results

Table 1 shows the average image quality metric scores. PAPR consistently outperforms the baselines across all metrics in both synthetic and real-world settings, without relying on specific initialization. These results demonstrate PAPR's ability to render highly realistic images that accurately capture details using an efficient and parsimonious representation.

Table 1: Comparison of image quality metrics (PSNR, SSIM and LPIPS [54]) on the NeRF Synthetic dataset [22] and Tanks & Temples subset [13]. Higher PSNR and SSIM scores are better, while lower LPIPS scores are better. All point-based baselines use a total number of 30 , 000 points and are initialized using the original techniques proposed by their respective authors. Despite being initialized from scratch, our method outperforms all baselines on all metrics for both datasets with the same total number of points.

4.2 Qualitative Results

Figure 5 shows a qualitative comparison between our method, PAPR, and the baselines on the NeRF Synthetic dataset. PAPR produces images with sharper details compared to the baselines. Notably, our method captures fine texture details on the bulldozer's body, the reflections on the drums and the crash, the mesh on the mic, and the reflection on the material ball. In contrast, the baselines either fail to capture the texture, exhibit blurriness in the generated output, or introduce high-frequency noise. These results validate the effectiveness of PAPR in capturing fine details and generating realistic renderings compared to the baselines. For additional qualitative results, please refer to the appendix.

4.3 Ablation Study

Effect of Number of Points We analyze the impact of the number of points on the rendered image quality in our method. The evaluation is performed using the LPIPS metric on the NeRF synthetic dataset. As shown in Figure 6, our method achieves better image quality by using a higher number of points. Additionally, we compare the performance of our model to that of DPBRF and Point-NeRF. The results show that our model maintains higher rendering quality even with a reduced number of

Lego

Drums

Mic

Materials

NeRF [22]

NPLF [24]

DPBRF [53]

SNP [56]

NeRF [48]

Splat-

Ground

Truth

ting [11]

points, as few as 1,000 points, compared to the baselines. These findings highlight the effectiveness of our method in representing the scene with high quality while using a parsimonious set of points.

Effect of Ray-dependent Point Embedding Design We analyze our ray-dependent point embedding design by gradually removing components, including the point position p i and the displacement vector t i,j , which are described in Sec. 3.3. Figure 7 shows both the qualitative and quantitative results of this analysis. Removing p i in key features introduces increased noise in the learnt point cloud, indicating its importance. Likewise, removing t i,j in both key and value features results in a failure to learn the correct geometry. These findings validate the effectiveness of our point embedding design and highlight the necessity of both the point position and displacement vector for achieving accurate and high-quality results.

4.4 Practical Applications

Zero-shot Geometry Editing We showcase the zero-shot geometry editing capability of our method by manipulating the positions of the point representations only . In Figure 8a, we present three cases that deform the object geometry: (1) rigid bending of the ficus branch, (2) rotation of the statue's head, and (3) stretching the tip of the microphone to perform a non-volume preserving transformation. The results demonstrate that our model effectively preserves surface continuity after geometry editing, without introducing holes or degrading the rendering quality. Additional results and comparisons with point-based baselines can be found in the appendix.

Object Manipulation In Figure 8b, we present object duplication and removal scenarios. Specifically, we add an additional hot dog to the plate and perform removal operations on some of the balls in the materials scene while duplicating others. These examples demonstrate the versatility of our method in manipulating and editing the objects in the scene.

Texture Transfer We showcase the capability of our method to manipulate the scene texture in Figure 8c. In this example, we demonstrate texture transfer by transferring the associated feature vectors from points corresponding to the mustard to some of the points corresponding to the ketchup. The model successfully transfers the texture of the mustard onto the ketchup, resulting in a realistic and coherent texture transformation. This demonstrates the ability of our method to manipulate and modify the scene texture in a controlled manner.

Exposure Control We showcase the ability of our method to adjust the exposure of the rendered image. To accomplish this, we introduce an additional random latent code input to our feature

renderer f θ R and finetune the model using the cIMLE technique [16]. Figure 8d shows different exposures in the rendered image by varying the latent code input. For more detailed information on this application, please refer to the appendix. This demonstrates our method's flexibility in controlling exposure and achieving desired visual effects in the rendered images.

1k

10k

20k

30k

0

.

02

0

.

1

0

.

2

Number of points

LPIPS-vgg

Ours

Point-NeRF

DPBRF

5 Discussion and Conclusion

Limitation Our pruning strategy currently assumes the background image to be given, making it suitable for scenarios with a near-constant background colour. However, this assumption may not hold in more diverse and complex backgrounds. To address this limitation, we plan to explore learning a separate model dedicated to handling background variations in future work.

Original

After Editing

(a) Deformation

(b) Duplication & Removal

(c) Texture

Transfer

(d) Exposure

Control

Societal Impact Expanding our method's capacity with more points and deeper networks can improve rendering quality. However, it's important to consider the environmental impact of increased computational resources, potentially leading to higher greenhouse gas emissions.

Conclusion In this paper, we present a novel point-based scene representation and rendering method. Our approach overcomes challenges in learning point cloud from scratch and effectively captures correct scene geometry and accurate texture details with a parsimoinous set of points. Furthermore, we demonstrate the practical applications of our method through four compelling use cases.

Acknowledgements This research was enabled in part by support provided by NSERC, the BC DRI Group and the Digital Research Alliance of Canada.

Rendered

Image

Depth

Point

Cloud

Rendered

Image

Depth

Point

Cloud

w/o

p

i

w/o

p

i

,

t

i,j

LPIPS:

LPIPS:

0.027

L00'0

LPIPS:

LPIPS:

0.034

110'0

LPIPS:

LPIPS:

0.356

0.126

References

Kara-Ali Aliev, Dmitry Ulyanov, and Victor S. Lempitsky. Neural point-based graphics. In European Conference on Computer Vision , 2019.

Mario Botsch, Alexander Sorkine-Hornung, Matthias Zwicker, and Leif P. Kobbelt. Highquality surface splatting on today's gpus. Proceedings Eurographics/IEEE VGTC Symposium Point-Based Graphics, 2005. , pages 17-141, 2005.

Jen-Hao Rick Chang, Wei-Yu Chen, Anurag Ranjan, Kwang Moo Yi, and Oncel Tuzel. Pointersect: Neural rendering with cloud-ray intersection. ArXiv , abs/2304.12390, 2023.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision , 2022.

John Flynn, Michael Broxton, Paul E. Debevec, Matthew DuVall, Graham Fyffe, Ryan S. Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2362-2371, 2019.

Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien P. C. Valentin. Fastnerf: High-fidelity neural rendering at 200fps. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 14326-14335, 2021.

Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T. Freeman. Unsupervised training for 3d morphable model regression. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8377-8386, 2018.

Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul E. Debevec. Baking neural radiance fields for real-time view synthesis. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5855-5864, 2021.

Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey. ArXiv , abs/2006.12057, 2020.

Hiroharu Kato, Y. Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3907-3916, 2017.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4), 2023.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics , 36(4), 2017.

Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis. Point-based neural rendering with per-view optimization. Computer Graphics Forum , 40, 2021.

Christoph Lassner and Michael Zollhöfer. Pulsar: Efficient sphere-based neural rendering. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1440-1449, 2021.

Ke Li*, Shichong Peng*, Tianhao Zhang*, and Jitendra Malik. Multimodal image synthesis with conditional implicit maximum likelihood estimation. International Journal of Computer Vision , May 2020.

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. ArXiv , abs/2007.11571, 2020.

Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7707-7716, 2019.

Matthew Loper and Michael J. Black. Opendr: An approximate differentiable renderer. In European Conference on Computer Vision , 2014.

Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, D. Salesin, William T. Freeman, and Michael Rubinstein. Layered neural rendering for retiming people in video. ACM Transactions on Graphics (TOG) , 39:1 - 14, 2020.

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. arXiv: Computer Vision and Pattern Recognition , 2019.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ArXiv , abs/2003.08934, 2020.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (TOG) , 41:1 15, 2022.

Julian Ost, Issam Hadj Laradji, Alejandro Newell, Yuval Bahat, and Felix Heide. Neural point light fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18398-18408, 2021.

Keunhong Park, U. Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5845-5854, 2020.

Songyou Peng, Chiyu Max Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. Shape as points: A differentiable poisson solver. In Neural Information Processing Systems , 2021.

Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or. Pix2vex: Imageto-geometry reconstruction using a smooth differentiable renderer. ArXiv , abs/1903.11149, 2019.

Hanspeter Pfister, Matthias Zwicker, Jeroen van Baar, and Markus H. Gross. Surfels: surface elements as rendering primitives. Proceedings of the 27th annual conference on Computer graphics and interactive techniques , 2000.

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10313-10322, 2020.

Ruslan Rakhimov, Andrei-Timotei Ardelean, Victor S. Lempitsky, and Evgeny Burnaev. Npbg++: Accelerating neural point-based graphics. ArXiv , abs/2203.13318, 2022.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 14315-14325, 2021.

Liu Ren, Hanspeter Pfister, and Matthias Zwicker. Object space ewa surface splatting: A hardware accelerated approach to high quality point rendering. Computer Graphics Forum , 21, 2002.

Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. A versatile scene model with differentiable visibility applied to generative pose estimation. 2015 IEEE International Conference on Computer Vision (ICCV) , pages 765-773, 2015.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. ArXiv , abs/1505.04597, 2015.

Darius Rückert, Linus Franke, and Marc Stamminger. Adop: Approximate differentiable one-pixel point rendering. ACM Trans. Graph. , 41:99:1-99:14, 2021.

Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 175-184, 2019.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5449-5459, 2021.

Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. ACM SIGGRAPH 2022 Conference Proceedings , 2022.

Towaki Takikawa, Joey Litalien, K. Yin, Karsten Kreis, Charles T. Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11353-11362, 2021.

Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. ArXiv , abs/2006.10739, 2020.

Anju Tewari, Otto Fried, Justus Thies, Vincent Sitzmann, S. Lombardi, Z Xu, Tanaba Simon, Matthias Nießner, Edgar Tretschk, L. Liu, Ben Mildenhall, Pranatharthi Srinivasan, R. Pandey, Sergio Orts-Escolano, S. Fanello, M. Guang Guo, Gordon Wetzstein, J y Zhu, Christian Theobalt, Manju Agrawala, Donald B. Goldman, and Michael Zollhöfer. Advances in neural rendering. Computer Graphics Forum , 41, 2021.

Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. arXiv: Computer Vision and Pattern Recognition , 2019.

Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Hang Yu, Wei Liu, X. Xue, and YuGang Jiang. Pixel2mesh: 3d mesh model generation via image guided deformation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43:3600-3613, 2020.

Yifan Wang, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG) , 38:1 - 14, 2019.

Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7465-7475, 2019.

Xiuchao Wu, Jiamin Xu, Zihan Zhu, Hujun Bao, Qixing Huang, James Tompkin, and Weiwei Xu. Scalable neural indoor scene rendering. ACM Transactions on Graphics (TOG) , 41:1 - 16, 2022.

Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum , 41, 2021.

Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5428-5438, 2022.

Bangbang Yang, Chong Bao, Junyi Zeng, Hujun Bao, Yinda Zhang, Zhaopeng Cui, and Guofeng Zhang. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. ArXiv , abs/2207.11911, 2022.

Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5491-5500, 2021.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5732-5741, 2021.

Yu-Jie Yuan, Yang tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: Geometry editing of neural radiance fields. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18332-18343, 2022.

Qian Zhang, Seung-Hwan Baek, Szymon Rusinkiewicz, and Felix Heide. Differentiable pointbased radiance fields for efficient view synthesis. SIGGRAPH Asia 2022 Conference Papers , 2022.

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 586-595, 2018.

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ArXiv , abs/1805.09817, 2018.

Yiming Zuo and Jia Deng. View synthesis with sculpted neural points. ArXiv , abs/2205.05869, 2022.

Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, and Markus H. Gross. Surface splatting. Proceedings of the 28th annual conference on Computer graphics and interactive techniques , 2001.

A Implementation Details

A.1 Proximity Attention

As introduced in Sec. 3.3, our model utilizes an attention mechanism to determine the proximity between the ray r j and a point i . To achieve this, we use three independent embedding MLPs to compute the key k i,j , value v i,j and query q j for the attention mechanism. The architecture details of these three embedding MLPs are shown in Figure 9. Conv:32→128,kernel size=3

Conv:128→256,kernel size=3

dj

Si,j

ti,j

PiE Pj

Si,j

tij

ReLU

√

Positional Encoding: L=6

Positional Encoding: L=6

Positional Encoding: L=6

feQ

fquery

fθk

rtey

fθv

ui

Conv:256→512,kernel size=3

√

ReLU

Layer Norm

Layer Norm

Linear: d →256

Transposed Conv:512→256,

Linear: dq →256

Linear:dk→256

ReLU

kernel size=2, stride=2

ReLU

ReLU

Linear:256→256

x4

x4

x6

Conv: 512→256,kernel size=3

Linear:256→256

Linear:256→256

ReLU

ReLU

Layer Norm

Layer Norm

Linear:256→32

Transposed Conv: 256→128,

Linear:256→256

Linear:256→256

kernel size=2, stride=2

√

Query qj

Key ki,j

Conv:256→128,kernel size=3

ReLU

Linear:128→3

A.2 Point Feature Renderer

As described in Sec. 3.3, our point feature renderer uses a modified U-Net architecture to generate colour outputs from the aggregated feature map. The architecture details are shown in Figure 10.

A.3 Point Pruning

As described in Sec. 3.4, we use a background token b to determine whether a ray intersects with the background. In our experiments, we set b = 5 . Moreover, we modify the calculation of the attention weights w i,j for each point in Eqn. 6 to incorporate both the background token and the influence score, the updated definition is as follows:

Here, a i,j is defined in Eqn. 6.

Figure 11 shows a comparison between the point cloud learnt by the model with and without pruning on the Lego scene. The results demonstrate that the adopted pruning strategy yields a cleaner point cloud with fewer outliers.

A.4 Point Growing

As mentioned in Sec. 3.4, our method involves growing points in the sparser regions of the point cloud. To identify these sparse regions, we compute the standard deviation σ i of the distances between

w/o pruning

w/ pruning

Reference

Feature Clusters

Caterpillar

Materials

Drums

each point i and its top 10 nearest neighbours. Subsequently, we insert new points near the points with higher values of σ i . The location of the new point near point i is selected as a random convex combination of i and its three nearest neighbours.

A.5 Clustering Feature Vectors

To demonstrate the effectiveness of our learnt point feature vectors, we visualize the feature vector clustering in Fig.12. The clustering is obtained by applying the K-Means clustering algorithm with K = 6 to the learnt feature vectors of the points. The resulting clustering reveals distinct separation, indicating the successful encoding of local information at surfaces with different textures and geometries.

More specifically, the feature vectors capture various colours within the scene. In the caterpillar scene, for instance, the point features associated with gray-coloured areas, such as the wheels and the side of the shovel, are clustered together. Moreover, the feature vectors can effectively differentiate between different materials, even when their colours may appear similar. For instance, in the materials scene, two balls on the bottom right most side have similar colours but different finishes-one with a matte appearance and the other with a polished look-and their feature vectors accurately distinguish between them. Additionally, the feature vectors demonstrate the capability to discern

variations in local geometry. In the drum scene, for example, the feature vectors for the large drum in the foreground differ from those representing the smaller drums in the background. These results highlight the encoding of distinct local scene information by the feature vectors, presenting potential utility in applications such as part segmentation.

A.6 Zero-shot Geometry Editing and Object Manipulation

As mentioned in Sec.4.4, our model is capable of zero-shot geometry editing and object manipulation by modifying the scene point cloud. For zero-shot geometry editing, we achieve this by altering the positions of the points, while for object addition or removal, we duplicate or delete specific points accordingly. It is important to note that our approach eliminates the need for any additional pre- or post-processing steps for rendering the scene after the editing process. This shows the simplicity and robustness of our approach in editing the scenes.

A.7 Texture Transfer

As mentioned in Sec. 4.4, to transfer textures between different parts of a scene, we first select the points representing the source and target parts, along with their corresponding point feature vectors. To capture the essential texture information, we identify the top K principal components for both the source and target feature vectors. To facilitate texture transfer, we project the source feature vectors onto the target principal components and get the corresponding coordinates. Finally, we apply an inverse transformation to bring these coordinates back to the original space along the source principal components. To render an image with transferred textures during testing, we simply replace the texture vectors of the source points with the transferred texture vectors. The remaining components of the model remain unchanged during testing.

FeatureMap

Z

Linear:128→256

ReLU

Linear:256→256

x6

ReLU

Linear:256→64

Max(0, x) + 1

β

Feature Map

×

+

β

UNet

A.8 Exposure Control

As exposure level is an uncontrolled variable in novel view synthesis of the scene, there can be multiple possible exposure levels for rendering unseen views not present in the training set. Therefore, we introduce an additional latent variable z to the model to explain this variability. Specifically, we modify the architecture of the point feature renderer f θ R and use the latent code z ∈ R 128 as an additional input. The latent code is first passed through an MLP, which produces a scaling vector γ ∈ R 32 and a translation vector β ∈ R 32 for the feature map. The scaling vector γ and the translation vector β are applied channel-wise to the feature map. The output is then fed into the U-Net architecture (described in Sec. A.2) to produce the final RGB image. The architecture details are shown in Figure 13.

To effectively train a model to solve this one-to-many prediction problem, we employ a technique known as cIMLE (conditional Implicit Maximum Likelihood Estimation) [16]. cIMLE is specifically useful for addressing the issue of mode collapse and effectively capturing diverse modes within the target distribution. Algorithm 1 shows the pseudo code of cIMLE:

Algorithm 1 Conditional IMLE Training Procedure

In our specific context, x i is the feature map, T θ is the modified point feature renderer, y i is the target RGB image. We first load pre-trained weights for all model parameters except the additional MLP for the latent variable before training. During test time, we randomly sample latent codes to change the exposure of the rendered image.

B Additional Results

B.1 Qualitative Comparison

We include extra qualitative results for NeRF Synthetic dataset [22] in Figure 14. Furthermore, we include qualitative comparisons between our method and the more competitive baselines, based on metric scores, on the Tanks & Temples subset in Figure 15.

ting [11]

Chair

Ficus

Hotdog

Ship

NeRF [22]

NPLF [24]

DPBRF [53]

SNP [56]

NeRF [48]

Splat-

Ground

Truth

Caterpillar

Family

Ignatius

Truck

NeRF [22]

SNP [56]

Splatting [11]

Ground Truth

INCELEE

QUARE

B.2 Point Cloud Learning

We show the point clouds generated by our method for each scene, depicted in Figure 16 for the NeRF Synthetic dataset [22], and Figure 17 for the Tanks & Temples [13] subset. These results demonstrate the effectiveness of our method in learning high-quality point clouds that correctly capture the intricate surfaces of the scenes.

Reference

Chair

Drums

Ficus

Hotdog

Lego

Materials

Mic

Ship

Reference

Caterpillar

Family

Ignatius

Truck

M

B.3 Zero-shot Geometry Editing

We present comparisons with point-based baselines on two scenes in which we apply non-volume preserving stretching transformations. In the first scene, we stretch the back of the chair, and in the second case, we stretch the tip of the microphone. As shown in Figure 18, NPLF [24], DPBRF [53], Point-NeRF [48] and Gaussian Splatting [11] either create holes or produce significant noise after the transformation. Additionally, SNP [56] fails to preserve the texture details following the edits, such as the golden embroidery pattern on the back of the chair and the mesh grid pattern on the tip of the microphone. In contrast, our method successfully avoids creating holes and effectively preserves the texture details after the transformation.

Before

After

Before

After

NPLF [24]

DPBRF [53]

SNP [56]

Point-NeRF [48]

Splatting [11]

B.4 Quantitative Results

We provide the metric scores broken down by scene on both datasets. Table 2 shows the per-scene scores for the NeRF Synthetic dataset [22] and Table 3 shows the scores for the Tanks & Temples [13] subset.

B.5 Ablation Study

We provide additional quantitative results for the ablation study. Figure 19 shows the PSNR and SSIM scores for the methods using different number of points. Additionally, Table 4 shows the PSNR and SSIM scores for the various choices of the ray-dependent point embedding. These results validate the effectiveness of our proposed method.

1k

10k

20k

30k

10

20

30

Number of points

PSNR

Ours

Point-NeRF

DPBRF

1k

10k

20k

30k

0

.

7

0

.

8

0

.

9

1

Number of points

SSIM

Ours

Point-NeRF

DPBRF

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a neural point renderer that improves the existing differentiable point rasterizers using a learned attention block, which improves the gradient behavior alleviating the \"vanishing gradient\" issue that comes with many existing differentiable point rasterizer. The method compares favorably with prior point-based NeRFs, jointly optimizing the point features and point locations, and achieving impressive 2D visual and 3D reconstruction quality even when using only poorly initialized sparse points. Additionally, it demonstrates useful applications thanks to the explicit point representation.\n\nSTRENGTHS:\n- The proposed method uses a learned attention block to determine the blending weights. It engineers the query, value and key embeddings using suitable geometric entities, which seems oddly effective (see my questions below) to emulate the similar effect of a fully deterministic point rasterizer in terms of the blending behavior w.r.t. point relative distance and depth.\n- The learning is further assisted with progressive point growing and pruning, which, as demonstrated in the ablation experiments, are effective techniques to achieve high quality reconstruction.\n- The visualization of point features through clustering is very interesting and insightful.\n- The paper is well-written, carefully explained, and the proposed components are thoroughly evaluated.\n- The improvement over existing point-based NeRF papers are significant.\n\nWEAKNESSES:\n1. No discussion about complexity. This is important, since the proposed method boasts to be able to compute gradient for all points regardless of their absolution distance to the target pixel in the pixel-space. This requires computing the attention with *every point* for every ray. Existing methods using RBF or other definition rasterization gradient (e.g. [44]) utilize handcrafted rules to reduce the impact of a pixel to a point as the point-pixel increases. While these rules may have the vanishing gradient problem the authors identified, they can serve as hard constraint to limit the number of target pixels required to evaluate the gradient for a specific point, i.e. pixels outside a search window can be ignored. With the proposed approach, and the claimed benefit, that each point receives gradient from all pixels, the computation is significantly higher. It's therefore important to breakdown the algorithm complexity of each step.\n\n1 (cont). Besides the theoretical complexity analysis, the authors should also provide the training and inference speed compared to a standard NeRF that achieves similar visual quality.\n\n2. the remark to [44] is not correct. [44] *is* able to render colored and shaded points. Besides, SynSin [Wiles et.al. CVPR 2020], Pulsar [Lassner, Christoph, and Michael Zollhofer. CVPR 2021] and Adop [35] are three relevant papers that should be cited too. In addition, I would argue that [44], Pulsar and [35] are more relevant than the baselines that are based on MVS initialization, and should be compared too.\n\n3. The necessity to use a convolutional U-Net in the end is disappointing, as the convolution is likely to induce flickering and other inconsistencies when moving the camera, which could be the cause of the flickering on drum surface in the video?\n\nQUESTIONS:\nI'd like the authors to address my concerns in the \"weakness\" section. Besides, please also help me understand the following questions in the rebuttal.\n\n1. Since the relative distance is completely learned without any prior, why would the network learn to produce higher weights for closer points. Could you visualize the learned weights for some sampled rays? Please also clarify whether Fig 3c is computed from a real learned result or plotted as the \"desired behavior\"? In addition, why do the curves go up again when the points move further away from the pixel (the right-most part of the blue curve and the left-most part of the orange curve).\n\n1. (cont) The inputs to the key and query embeddings don't share any similarity at all. The MLPs need to learn to map them to compatible space and ensure that keys (3) of points that are close to the rays (which would correspond to the Fig 3c gradient behavior) are more similar to the query (5). How does the network learn this without any prior? Do you need special initialization for the MLPs? And what is the convergence rate.\n\n2. Ideally the learnable foreground score should be related depth and occlusion. But without constraint, this learned foreground score can be abused to label \"outlier points\" that are far from any target pixels due to optimization problems, which can be conveniently pruned and therefore ignored in the optimization. Please visualize the foreground score before point pruning to ensure that this score indeed correspond to visibility.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel view synthesis and 3D reconstruction framework via a point-based representation and a jointly learned transformer-based differentiable renderer. Given multi-view images of a foreground-centric scene, the method can effectively deform a point-cloud from scratch to match the surfaces, with an optional pruning and growing mechanism. Each point is associated with a learned feature, which allows novel views of the scene to be rendered through a transformer-based point renderer.\n\nExperiments show that this method leads to high-quality geometry and novel view synthesis results, significantly outperforming existing point-based NVS methods as well as the vanilla NeRF.\n\nSTRENGTHS:\n### S1 - A solid pipeline for point-based reconstruction and rendering\n- The proposed pipeline is very well designed and validated. In particular:\n    - The design of the transformer-based point renderer is interesting. Ray direction is treated as the query feature, point position the key, and point features the value, which make a lot of sense.\n    - The transformer directly outputs the weights of each point for integration, without a separate volume rendering function as in Point-NeRF that would introduce another order of computation.\n\n### S2 - Great results on geometry reconstruction and novel view synthesis\n- The method effectively deforms a point cloud from a initial sphere to complex surfaces, which is a very compact representation of the scene geometry, and allows for various scene editing.\n- The NVS results are also quite impressive, even outperforming NeRF (although I guess one might be able to further tune NeRF to achieve even better numbers).\n\n### S3 - Good demonstrations of scene editing with points\n- The paper presents a number of scene editing applications, including deformation, object removal and appearance editing. Having an explicit point-based representation makes such editing much easier than implicit NeRF/volumes.\n\n### S4 - Good presentation\n- The paper is well written, with clear motivation and illustrative figures.\n\nWEAKNESSES:\n### W1 - Renderer seems to be learned jointly on one scene only\n- In theory, the transformer-based renderer should be generic and can be shared across any scene?\nDoes it generalize to other scenes?\n\n### (minor) W2 - Technical details\n- There seems no explicit inductive bias in the renderer that would encourage it to actually put higher weights to points closer to the ray and to the camera (within the K nearest points). Does the learned renderer actually utilizes the information of $\\mathbf{s}_{i,j}$ and $\\mathbf{t}$ as we expect? It would be helpful to visualize the weights.\n- Sup. mat. Sec. B.3 Eq (10): Is the $w_{i,j}$ here taken before or after the softmax in Eq. (6)?\n    - If before, $w_{i,j}$ is not guaranteed to be positive, and Eq. (7) and Eq. (10) would be problematic, as both $w_{i,j}$ and $\\tau_i$ can be either positive or negative.\n    - If after, there would be double softmax / exponential. Does that cause any issue, eg. vanishing gradients?\n- Pruning points with $\\tau_i \\lt 0$ seems quite arbitrary. I suppose this is related to the choice of the predefined background weight scalar $b$. Is the threshold $0$ chosen empirically? (why not, say, $\\tau_i \\lt 1$ or $\\tau_i \\lt -2.64$?)\n- What is the run time? How does it compare to say NeRF?\n- What happens if it applied to forward facing scenes with complex background, eg. LLFF? The limitation discusses the failure in modeling background in 360° scenes, but what about forward-facing scenes, where points can be legitimately assigned to \"background\"?\n\n### Additional comments\n- Fig. 4: When the pruning is disabled, it is not surprising at all that Point-NeRF fails to represent the surface using the points, as it is still using a volume rendering function where the underlying geometry is ultimately still represented by the density values. Although, I agree it is great that the proposed method is able to deform the points correctly around the surface without pruning, thanks to the distance-aware weighting in the rendering function I suppose.\n- Only one attention layer?\n- Any points remain inside the object?\n- Line 109: minor grammatical issues.\n\nQUESTIONS:\nNo critical concerns. It would be helpful to elaborate on the generalization of the jointly learned transformer renderer.\n\nLIMITATIONS:\nThe paper includes a nice discussion on the limitation regarding background modeling.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method that consists of a point-based scene representation and a differentiable renderer. The proposed representation utilizes relative distance to characterize the contribution of each point, preventing the vanishing gradient issue. The method outperforms baselines in terms of optimizing point cloud positions, rendering. And it enables interesting applications.\n\nSTRENGTHS:\n- Overview figure is clear, well-conveying the idea of the paper.\n\n- The representation is simple, reasonable and well-desinged.\n\n- The proposed method outperforms the baselines.\n\nWEAKNESSES:\nThe paper is sound in general. However there are several points that can be clarified.\n\n- Feature vector: learned or deterministically determined? If it is the latter case, how it can be determined?\n\n- In the figure 3-(c), why does the gradient scale increases again when the target pixel position deviates much from the point cloud?\n\n- In L148-151, it is mentioned that using the relative distance guarantess that there are always some points contributing to each pixel, even if their absolute distances are large.\nHowever, if the distance is too far, wouldn't this design enlarge the error, i.e., generating the color of the unrelated point cloud? I think setting a threshold would be helpful to prevent error, or is this thresholding same as the filtering process in L164?. Also, if the filtering is applied (L164), wouldn't the points that are far away be removed and therefore cannot contribute to the pixel? And how is this filtering threshold decided? Would the value of it affect the performance?   \n\n- How can the view dependent effect be modeled? From my understanding, different viewing direction results in different s,t value of each point, and thus it is possible to model it?\n\n- What is the fixed background token b in L189?\n\n- In Figure 5, the result of proposed method on the Mic sample shows very fine-grained detail similar to GT. However, the result on the Materials sample is relatively blurry.\nWhat is the reason behind the quality difference?\n\n- It would be great if the visual ablations can be done when using the different number of points.\n\nQUESTIONS:\n- How can the transparent material modeled with the proximity attention?\n\n- How this approach can be extended to complicated scene with lot of occlusions?\n\n- This approach can work on rigid deformation for some extent. How can this approach be extended to non-rigidly deforming scene?\n\nLIMITATIONS:\n- Clarifying some details (as mentioned in the weakness section) could further improve the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel point-based rendering approach, PAPR, to learn a point cloud for neural rendering under the supervision of posed RGB images. At the core of softening the gradient flow by using an attention mechanism in a novel and elegant way, the proposed PAPR enables high-fidelity rendering while explicitly learning the parsimonious point-based scene representation from scratch. Based on this, the proposed PAPR enjoys flexibility for geometry editing, object manipulation, texture transfer, and exposure control, all with impressive results.\n\nSTRENGTHS:\n1. Learning a parsimonious point cloud from scratch in the form of neural rendering without using the 3D scene information is very interesting and has many potential applications. \n\n2. The core module, proximity attention, is well-designed and should be computationally efficient. Instead of sampling points along rays, the proximity attention module proposed in this paper uses the projection relationship between rays and points to aggregate the proximate points in an attention form and feed the point features in a U-Net for rendering. The overall pipeline is neat and technically sound. The computation should be efficient as well.\n\nWEAKNESSES:\n1. Because the authors emphasize the parsimony property of the point cloud, the number of points should be the key to be studied. Although the authors have provided an ablation study in Fig. 6 on the effect of the number of points on rendered image quality, some in-depth discussion would further strengthen this paper. Because the authors mentioned that the proposed method is currently suitable for the near-constant background color, I am curious about the parsimony of the point cloud concerning the non-constant background color. \n\n2. The growing scheme is somewhat heuristic with predefined hyperparameters such as 500 iterations, top-10 nearest neighbors, etc. There should be some ablation study for the effect of these hyperparameters. Furthermore, because proximity attention could capture the relationship between points, is there a way to use the learned affinity to identify the sparse regions?\n\nQUESTIONS:\nBecause this paper has shown their ability to learn a parsimonious point cloud from scratch, what will be if we have some inductive initialization of the point cloud? For example, could the proposed method refine the sparse reconstruction quality if we have a sparse reconstruction result for the scene by COLMAP? I am also curious about the potential of doing bundle adjustment with the proposed PAPR.\n\nLIMITATIONS:\nThe authors have discussed their limitations. I don't have any concern on this point.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of point-based rendering and proposes a new point-based differentiable rendering pipeline with proximity attention. With an attention-based point rendering, the proposed pipeline avoids the tuning of points' radius or splatting sizes that typical point rendering requires. Such a design enables efficient and high-quality rendering. The effectiveness is verified on NeRF synthetic and Tank and Temple datasets.\n\nSTRENGTHS:\n1. The proposed proximity attention is neat and effective.\n\n1. The paper is well-written and easy to follow.\n\n2. The experiments are solid.\n\nWEAKNESSES:\n### 1. Choices of query, key, value for attention\n\na. I think the paper misses an important ablation of various representations for query, key and value for the proximity attention. Currently, query is represented as ray direction, key is used as $(t, s, \\mathbf{p})$, and value is represented as $(t, s, \\mathbf{u})$. A natural question is what if 1) we remove $s$ in both key and value; 2) we have key and value are both represented as $(t, s, \\mathbf{p}, \\mathbf{u})$ (adding feature vector)?\n\nb. What is related is Fig. 7. Can authors clarify: for \"incrementally remove key components in the point embedding\" (Fig. 7's caption), does it mean removing $t_{i,j}$ in both key and value representation?\n\n### 2. Relationship to [24]\n\na.  I feel like the proximity attention is quite similar/related to the **Ray-centric Point Encoding** in [24]. Essentially, [24] also proposes an attention-based point embedding, where \"query\" is the ray direction same as this paper; \"key\" and \"value\" are concatenations from points's positions, distance between point and ray, angle between points and ray (also similar to the paper).\n\nb. Therefore, to better position the paper, I want to understand the difference between this paper and [24]. This is one of the major motivations why I am curious about the ablations about the choices of query/key/values.\n\nc. One difference I can see is that [24] uses an MLP as the renderer while PAPR uses a UNet. Can the author provide an ablation where an MLP is used for $f_{\\theta_\\mathcal{R}}$ (L181)? If PAPR does not work well with MLP, then it seems like the major improvement may come from the renderer's capabilities instead of the proposed attention-based point rendering.\n\nd. Another difference is that [24] uses an image-based feature as the point's feature. Such a design does not allow for changing points's positions if the points's features are not updated later. For Fig. 4, authors state \"we made modifications to their official code to enable point position optimization\" (L214-215). Can the authors clarify whether the features will be updated or not? If the features are not updated, then this needs clarifications and it may be better to replace [24] results in Fig. 4 with initializing from a sparse point cloud of SfM or MVS. This will not shade the advantage of PAPR.\n\ne. Similarly to d above, do the results of [24] in Tab. 1 come from initializing from a sphere? If so, I think it is better to replace it with initializing from a sparse point cloud and give a note in the table about this.\n\n### 3. Speed\n\na. An advantage of traditional point-based rendering [15, 53] is the speed. Can authors also provide the inference speed of PAPR? \n\nb. What is related is the training speed. Can authors also provide the training time of PAPR?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe authors provide discussion of limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}