SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yanshu Zhang*, Shichong Peng*, Alireza Moazeni, Ke Li  
APEX Lab
School of Computing Science
{yanshu_zhang, shichong_peng, seyed_alireza_moazenipourasil, keli}@sfu.ca
Learning accurate and parsimonious point cloud representations of scene surfaces from scratch remains a challenge in 3D representation learning. Existing point-based methods often suffer from the vanishing gradient problem or require a large number of points to accurately model scene geometry and texture. To address these limitations, we propose Proximity Attention Point Rendering (PAPR), a novel method that consists of a point-based scene representation and a differentiable renderer. Our scene representation uses a point cloud where each point is characterized by its spatial position, influence score, and view-independent feature vector. The renderer selects the relevant points for each ray and produces accurate colours using their associated features. PAPR effectively learns point cloud positions to represent the correct scene geometry, even when the initialization drastically differs from the target geometry. Notably, our method captures fine texture details while using only a parsimonious set of points. We also demonstrate four practical applications of our method: zero-shot geometry editing, object manipulation, texture transfer, and exposure control. More results and code are available on our project website.
Learning 3D representations is crucial for computer vision and graphics applications. Recent neural rendering methods [22, 42, 50, 23, 15, 48] leverage deep neural networks within the rendering pipelines to capture complex geometry and texture details from multi-view RGB images. These methods have many practical applications, such as generating high-quality 3D models [18, 43, 3, 7] and enabling interactive virtual reality experiences [29, 25, 52]. However, balancing representation capacity and computational complexity remains a challenge. Large-scale representations offer better quality but demand more resources. In contrast, parsimonious representations strike a balance between efficiency and quality, enabling efficient learning, processing, and manipulation of 3D data.
3D representations can be categorized into volumetric and surface representations. Recent advances in volumetric representations [22, 48, 50, 37, 17] have shown impressive results in rendering quality. However, the cubic growth in encoded information as the scene radius increases makes volumetric representations computationally expensive to process. For example, to render a volumetric representation, all the information along a ray needs to be aggregated, which requires evaluating many samples along each ray. In contrast, surface representations are more parsimonious and efficient since the encoded information grows quadratically. So surface representations can be rendered with much fewer samples.
Efficient surface representations include meshes and surface point clouds. Meshes are difficult to learn from scratch, since many constraints (e.g., no self-intersection) need to be enforced. Additionally, the topology of the initial mesh is fixed, making it impossible to change during training [43]. In contrast,
*Denotes equal contribution
Figure 1: Our proposed method, PAPR, jointly learns a point-based scene representation and a differentiable renderer from scratch using only RGB supervision. PAPR effectively learns the point positions (a) that represent the correct surface geometry, as demonstrated by the depth map (b). Notably, PAPR achieves this even when starting from an initialization that substantially deviates from the target geometry. Furthermore, PAPR learns a view-independent feature vector for each point, capturing the local scene content. The clustering of point feature vectors is shown in (c). The renderer selects and combines these feature vectors to generate high-quality colour rendering (d).
point clouds offer more flexibility in modelling shapes with varying topologies. Hence, in this work, we focus on using point clouds to represent scene surfaces.
Learning a scene representation from scratch requires a differentiable renderer that can pass large gradients to the representation. Designing a differentiable renderer for point clouds is non-trivial - each point is infinitesimal and rarely intersects with rays, it is unclear what the output colour at such rays should be. Previous methods often rely on splat-based rasterization techniques, where points are turned into disks or spheres [45, 15, 35, 53, 11, 56]. These methods use a radial basis function (RBF) kernel to calculate the contribution of each point to each pixel. However, determining the optimal radius for the splats or the RBF kernel is challenging [9], and these methods suffer from the issue of vanishing gradient when the ray is far away from the points, limiting their ability to learn the ground truth geometry that drastically differs from the initial geometry. Additionally, the radius of each splat should be small for accurate texture modelling, and this would require a point cloud with a large number of points, which are difficult to process.
To address these limitations, we introduce a novel method called Proximity Attention Point Rendering (PAPR). PAPR consists of a point-based scene representation and a differentiable renderer. The scene representation is constructed using a point cloud, where each point is defined by its spatial position, an influence score indicating its influence on the rendering, and a view-independent feature vector that captures the local scene content. Our renderer learns to directly select points for each ray and combines them to generate the correct colour using their associated features. Remarkably, PAPR can learn accurate geometry and texture details using a parsimonious set of points, even when the initial point cloud substantially differs from the target geometry, as shown in Figure 1. We show that the proposed ray-dependent point embedding design is crucial for effective learning of point cloud positions from scratch. Furthermore, our experiments on both synthetic and real-world datasets demonstrate that PAPR outperforms prior point-based methods in terms of image quality when using a parsimonious set of points. We also showcase four practical applications of PAPR: zero-shot geometry editing with part rotations and deformations, object duplication and deletion, texture transfer and exposure control. In summary, our contributions are as follows:
1. We propose PAPR, a novel point-based scene representation and rendering method that can learn point-based surface geometry from scratch.
2. We demonstrate PAPR's ability to capture correct scene geometry and accurate texture details using a parsimonious set of points, leading to an efficient and effective representation.
3. We explore and demonstrate four practical applications with PAPR: zero-shot geometry editing, object manipulation, texture transfer and exposure control.
Our work focuses on learning and rendering 3D representation, and it is most related to differentiable rendering, neural rendering and point-based rendering. While there are many relevant works in these fields, we only discuss the most pertinent ones in this context and refer readers to recent surveys [9, 41, 47] for a more comprehensive overview.
Differentiable Rendering For 3D Representation Learning Differentiable rendering plays a crucial role in learning 3D representations from data as it allows gradients to be backpropagated from the rendered output to the representation. Early mesh learning approach, OpenDR [19], approximate gradients using localized Taylor expansion and differentiable filters, but do not fully leverage loss function gradients. Neural Mesh Renderer [10] proposes non-local approximated gradients that utilize gradients backpropagated from a loss function. Some methods [33, 18, 27] modify the forward rasterization step to make the pipeline differentiable by softening object boundaries or making the contribution of triangles to each pixel probabilistic. Several point cloud learning methods, such as differentiable surface splatting [44] and differentiable Poisson solvers [26], explore surface reconstruction techniques but they do not focus on rendering fine textures. In contrast, our method jointly learns detailed colour appearance and geometry representations from scratch.
Neural Scene Representation In recent years, there has been a growing interest in utilizing neural networks to generate realistic novel views of scenes. Some approaches utilize explicit representations like meshes [42, 49], multi-plane images [5, 20, 21, 36, 55], and point clouds [1, 30, 15]. Another approach represents the scene as a differentiable density field, as demonstrated by Neural Radiance Field (NeRF) [22]. However, NeRF suffers from slow sampling speed due to volumetric ray marching. Subsequent works have focused on improving training and rendering speed through space discretization [39, 6, 8, 4, 31, 51, 46] and storage optimizations [23, 38, 50, 37, 17]. These methods still rely on volumetric representations, which have a cubic growth in encoded information. In contrast, our proposed method leverages surface representations, which are more efficient and parsimonious.
Point-based Representation Rendering and Learning Rendering point clouds using ray tracing is challenging because they do not inherently define a surface, leading to an ill-defined intersection between rays and the point cloud. Early approaches used splatting techniques with disks, ellipsoids, or surfels [57, 32, 2, 28]. Recent methods [1, 30, 24, 14] incorporate neural networks in the rendering pipeline but assume ground truth point clouds from Structure-from-Motion (SfM), Multi-View Stereo (MVS) or LiDAR scanning without optimizing point positions. To learn the point positions, splat-based rasterization methods [45, 15, 35, 53, 11, 56] employ radial basis function kernels to compute point contributions to pixels but struggle with finding optimal splat or kernel radius [9] and suffer from gradient vanishing when points are far from target pixels. Consequently, these methods require a large number of points to accurately model scene details and are limited to learn small point cloud deformation. In contrast, our method can capture scene details using a parsimonious set of points and perform substantial deformation in the initial point cloud to match the correct geometry.
Another approach [3] predicts explicit intersection points between rays and optimal meshes reconstructed from point clouds, but it requires auxiliary data with ground truth geometry for learning and RGBD images for point cloud initialization. Our method overcomes these limitations. Xu et al. [48] represent radiance fields with points and use volumetric ray marching, our approach utilizes more efficient and parsimonious surface representations and generates output colour for each ray with only a single forward pass through the network.
3 Proximity Attention Point Rendering
Figure 2 provides an overview of our proposed end-to-end learnable point-based rendering method. The input to our model includes a set of RGB images along with their corresponding camera inclusions and extrinsics. Our method jointly learns a point-based scene representation, where each point comprises a spatial position, an influence score, and a view-independent feature vector, along with a differentiable renderer. This is achieved by minimizing the reconstruction loss between the rendered image  $\hat{\mathbf{I}}$  and the ground truth image  $\mathbf{I}_{gt}$  according to some distance metric  $d(\cdot ,\cdot)$ , w.r.t. the scene representation and the parameters of the differentiable renderer. Unlike previous point cloud learning methods [48, 45, 15, 35, 53, 11, 3, 56] which rely on multi-view stereo (MVS), Structure-fromMotion (SfM), depth measurements or object masks for initializing the point cloud positions, our method can learn the point positions from scratch, even when the initialization significantly differs from the target geometry as demonstrated in Figure 4. In the following sections, we provide a more detailed description of each component of our proposed method.
(a) Ray-dependent Point Embedding
(b) Proximity Attention
(c) Point Feature Rendering
Figure 2: An overview of the proposed pipeline for rendering a point-based scene representation, where each point is defined by its spatial position, an influence score, and a view-independent feature vector. (a) Given a ray, ray-dependent features are created for each point. These features are used to generate the key, value, and query inputs for a proximity attention model. (b) The attention model selects the points based on their keys and the query, and combines their values to form an aggregated feature. (c) The aggregated feature is gathered for all pixels to create a feature map. This feature map is then passed through a feature renderer, implemented using a UNet architecture, to produce the output image. The entire pipeline is jointly trained in an end-to-end manner using only supervision from the ground truth image.
3.1 Point-based Scene Representation
Our scene representation consists of a set of  $N$  points  $P = \{(\mathbf{p}_i, \mathbf{u}_i, \tau_i)\}_{i=1}^N$  where each point  $i$  is characterized by its spatial position  $\mathbf{p}_i \in \mathbb{R}^3$ , an individual view-independent feature vector  $\mathbf{u}_i \in \mathbb{R}^h$  that encodes local appearance and geometry and an influence score  $\tau_i \in \mathbb{R}$  that represents the influence of the point on the rendering.
Notably, we make the feature vector for each point to be independently learnable, thereby allowing the content encoded in the features to be independent of the distance between neighbouring points. As a result, after training, points can be moved to different positions without needing to make a corresponding change in the features. This design also makes it possible to increase the fidelity of the representation by adding more points, thereby capturing more fine-grained details. Additionally, we specifically choose the feature vectors to be view-independent, ensuring that the encoded information in the representation remains invariant to view changes.
3.2 Differentiable Point Rendering with Relative Distances
Image Plane
Figure 3: Comparison of point contributions in splat-based rasterization and the proposed method. Three points are projected onto the image plane, with the ground truth image shown in the background. The bar charts illustrate the contribution of each point towards a specific pixel, represented by the red star. (a) If the points are splats with hard boundaries, none of them intersect with the given pixel, resulting in zero contributions. Consequently, the gradient of the loss at the pixel w.r.t. each point is non-existent, hindering the reduction of loss at that pixel. (b) For splats with soft boundaries, although the point contributions are non-zero, they are extremely small, leading to vanishing gradients. (c) Our proposed method normalizes the contribution of all points to the given pixel. This ensures that there are always points with substantial contributions, enabling learning point positions from scratch.
(a) Hard Boundary Splats
(b) Soft Boundary Splats
(c) Proposed
To learn the scene representation from images from different views, we minimize the reconstruction loss between the rendered output from those views and the ground truth image w.r.t. the scene representation. Typically, we do so with gradient-based optimization methods, which need to compute
the gradient of the loss w.r.t. the scene representation. For this to be possible, the rendered output must be differentiable w.r.t. the representation – in other words, the renderer must be differentiable.
In classical rendering pipelines, point clouds are often rendered with splat-based rasterization. However, when attempting to learn point clouds from scratch using this method, several challenges arise for the following reasons. A splat only contributes to the pixels it intersects with, so when a splat does not intersect with a specific pixel, it does not contribute to the rendered output at that pixel. As a result, the gradient of the loss at that pixel w.r.t. the position of the splat is zero. Now, consider a pixel in the ground truth image that no splat intersects with. Even though the loss at that pixel might be high, the gradient w.r.t. the position of every splat is zero, so the loss at that pixel cannot be minimized, as illustrated in Figure 3a.
To address this issue, prior methods have employed a radial basis function (RBF) kernel to soften splat boundaries. This approach makes the contribution of each splat to each pixel non-zero. However, even with the soft boundaries, the contribution of a splat to a pixel remains small if the pixel is located far away from the centre of the splat. Consequently, for a pixel far away from the centres of all splats, the gradient of the loss at the pixel w.r.t. each splat remains too minuscule to exert a significant influence on its position, as shown in Figure 3b – in other words, the gradient vanishes. As a result, even though the loss at the pixel might be high, it will take a long time before it is minimized.
We observe that with both hard- and soft-boundary splats, the contribution of each point to a given pixel only depends on its own absolute distance to the pixel, without regard to the distances of other points. This gives rise to the vanishing gradient problem because the absolute distances for all points can be high. To address this issue, we make the contributions per point towards a pixel depend on relative distances rather than absolute distances. Instead of relying solely on the distance to the pixel from a single point to compute its contribution, we normalize the contributions across all points so that the total contribution from all points is always 1. This ensures that there are always points with significant contributions to each pixel, even if the pixel is far away from all points, as shown in Figure 3c.
This allows points to be moved far from their initial positions, so the initialization can differ substantially from the ground truth scene geometry. This also means that we can move sufficiently many points from elsewhere to represent the scene geometry faithfully. On the other hand, the learnt representation uses no more points than necessary to represent each local part of the scene geometry faithfully, because if there were extraneous points, they could be moved to better represent other parts. The end result is a parsimonious scene representation that uses just enough points to faithfully represent each part of the scene geometry. So, fewer points tend to be allocated to smooth parts of the surface and the interior of opaque surfaces, because having more points there would not improve the rendering quality significantly. Moreover, since the points can be far away from a pixel without having their contributions diminished, we can preserve the surface continuity even if we apply a non-volume preserving transformation (e.g., stretching) to the point cloud post-hoc.
3.3 Implementation of Relative Distances with Proximity Attention
We design an attention mechanism to compute contributions from relative distances, which we dub proximity attention. Unlike typical attention mechanisms, the queries correspond to rays and the keys correspond to points. To encode the relationship between points and viewing directions, we design an embedding of the points that account for their relative positions to rays. This embedding is then fed in as input to the proximity attention mechanism. We describe the details of each below.
Ray-dependent Point Embedding Given a camera pose  $\mathcal{C}$  specified by its extrinsics and intrinsics, and a sensor resolution  $H\times W$ , we utilize a pinhole camera model to cast a ray  $\mathbf{r}_j$  from the camera centre to each pixel. Each ray  $\mathbf{r}_j$  is characterized by its origin  $\mathbf{o}_j\in \mathbb{R}^3$  and unit-length viewing direction  $\mathbf{d}_j\in \mathbb{R}^3$ , both defined in the world coordinate system.
To obtain the ray-dependent point embedding of point  $\mathbf{p}_i$  with respect to ray  $\mathbf{r}_j$ , we start by finding the projection  $\mathbf{p}_i'$  of the point onto the ray:
$$
\mathbf {p} _ {i} ^ {\prime} = \mathbf {o} _ {j} + \left\langle \mathbf {p} _ {i} - \mathbf {o} _ {j}, \mathbf {d} _ {j} \right\rangle \cdot \mathbf {d} _ {j} \tag {1}
$$
where  $\langle \cdot ,\cdot \rangle$  denotes the inner product. Next, we compute the displacement vectors,  $\mathbf{s}_{i,j}$  and  $\mathbf{t}_{i,j}$ :
$$
\mathbf {s} _ {i, j} = \mathbf {p} _ {i} ^ {\prime} - \mathbf {o} _ {j}, \mathbf {t} _ {i, j} = \mathbf {p} _ {i} - \mathbf {p} _ {i} ^ {\prime}, \tag {2}
$$
Here,  $\mathbf{s}_{i,j}$  captures the depth of the point  $\mathbf{p}_i$  with respect to the camera centre  $\mathbf{o}_j$  along the ray, while  $\mathbf{t}_{i,j}$  represents the perpendicular displacement from the ray to the point  $\mathbf{p}_i$ . The final ray-dependent point embedding contains these displacement vectors,  $\mathbf{s}_{i,j}$  and  $\mathbf{t}_{i,j}$ , as well as the point position  $\mathbf{p}_i$ .
Proximity Attention For a given ray  $\mathbf{r}_j$ , we start by filtering the top  $K$  nearest points  $\mathbf{p}_i$  based on the magnitude of their perpendicular displacement vector  $||\mathbf{t}_{i,j}||$ . This allows us to identify the points that are most relevant for modelling the local contents of the ray.
We adopt the ray-dependent point embedding method described earlier for the  $K$  closest points, which serves as the input to construct the key in the attention mechanism. For the value branch input, we utilize the  $K$  associated point feature vectors  $\mathbf{u}_i$ , along with their displacement vectors  $\mathbf{s}_{i,j}$  and  $\mathbf{t}_{i,j}$ . For the input to the query branch, we use the ray direction  $\mathbf{d}_j$ . Additionally, we apply positional encoding [22, 40]  $\gamma(p) = [\sin(2^l\pi p), \cos(2^l\pi p)]_{l=0}^{L=6}$  to all components except for the feature vector. These inputs pass through three separate embedding networks  $f_{\theta_K}$ ,  $f_{\theta_V}$  and  $f_{\theta_Q}$ , which are implemented as multi-layer perceptrs (MLPs). The resulting outputs form the final key, value, and query for the attention mechanism:
$$
\mathbf {k} _ {i, j} = f _ {\theta_ {K}} \left(\left[ \gamma \left(\mathbf {s} _ {i, j}\right), \gamma \left(\mathbf {t} _ {i, j}\right), \gamma \left(\mathbf {p} _ {i}\right) \right]\right) \tag {3}
$$
$$
\mathbf {v} _ {i, j} = f _ {\theta_ {V}} \left(\left[ \gamma \left(\mathbf {s} _ {i, j}\right), \gamma \left(\mathbf {t} _ {i, j}\right), \mathbf {u} _ {i} \right]\right) \tag {4}
$$
$$
\mathbf {q} _ {j} = f _ {\theta_ {Q}} (\gamma (\mathbf {d} _ {j})) \tag {5}
$$
To compute the aggregated value for the query, we calculate the weighted sum of the values as  $\mathbf{f}_j = \sum_{i=1}^{K} w_{i,j} \mathbf{v}_{i,j}$ , where each weighting term is calculated by applying the softmax function to the raw attention scores  $a_{i,j}$ . The raw attention score  $a_{i,j}$  is obtained by applying a ReLU function to the dot product between the query  $\mathbf{q}_j$  and the keys  $\mathbf{k}_{i,j}$  scaled by their dimensionality  $\mu$ :
$$
w _ {i, j} = \frac {\exp \left(a _ {i , j}\right)}{\sum_ {m = 1} ^ {K} \exp \left(a _ {m , j}\right)}, \text {w h e r e} a _ {i, j} = \max  (0, \frac {\left\langle \mathbf {q} _ {j} , \mathbf {k} _ {i , j} \right\rangle}{\sqrt {\mu}}) \tag {6}
$$
By aggregating the features  $\mathbf{f}_j$  for all rays  $\mathbf{r}_j$  corresponding to each pixel on the image plane with resolution  $H\times W$ , we construct the feature map  $F_{\mathcal{C}}\in \mathbb{R}^{H\times W\times d_{\mathrm{feat}}}$  for a given camera pose  $\mathcal{C}$ .