SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Amine Ouasfi Adnane Boukhayma  
Inria, Univ. Rennes, CNRS, IRISA, M2S, France
Feedforward generalizable models for implicit shape reconstruction from unoriented point cloud present multiple advantages, including high performance and inference speed. However, they still suffer from generalization issues, ranging from underfitting the input point cloud, to misrepresenting samples outside of the training data distribution, or with topologies unseen at training. We propose here an efficient mechanism to remedy some of these limitations at test time. We combine the inter-shape data prior of the network with an intra-shape regularization prior of a Nystrom Kernel Ridge Regression, that we further adapt by fitting its hyperprameters to the current shape. The resulting shape function defined in a shape specific Reproducing Kernel Hilbert Space benefits from desirable stability and efficiency properties and grants a shape adaptive expressiveness-robustness trade-off. We demonstrate the improvement obtained through our method with respect to baselines and the state-of-the-art using synthetic and real data.
Enabling machines to understand and navigate 3D is a major challenge. Instances of this ability include downstream tasks in computer vision and graphics, such as shape reconstruction from noisy, incomplete and relatively sparse point clouds. Building full shapes from point clouds is all the more an important problem in account of the ubiquity of this light, albeit incomplete, 3D shape representation, whether it is acquired from the increasingly democratized consumer grade and industrial depth sensors or e.g. as obtained from photogrammetry (e.g. Structure From Motion, Multi-View Stereo [62, 63]). Whilst Classical optimization based approaches such as Poisson Reconstruction [33] and Moving least squares [27] can deliver mostly good reconstructions for all intents and purposes, they require still dense clean point sets with reliable normal estimations. More recently, deep learning based alternatives have been shown to offer faster and more robust predictions especially for noisy and sparse inputs, without requiring normal information.
Within this class of methods, state-of-the-art feed forward (test time optimization-free) generalizable models are based on implicit neural shape representations, where explicit meshes can be extracted at test time using Marching Cubes [43]. While they offer many desirable properties, such as all-round good performance and fast inference (within tens of milliseconds on a standard GPU), they still suffer from limitations, some of which we come across upon examining the state-of-the-art method [7] for instance:
Limited OOD generalization As these models are typically trained with full 3D supervision, training commonly uses synthetic data such as the ShapeNet [12] dataset, large real 3D supervision being famously hard to come by. This results in a performance drop for real test data outside the training distribution. Comparing Tables 2 and 6, the performance of [7] drops by roughly fourfold going from testing on synthetic data to real scans.
Limited input size generalization As illustrated in table 7, when presented with testing point clouds that are larger (e.g. 10k) than the training size (3k), the performance does not scale and even deteriorates compared to testing on the training size.
Underfitting the input There is no explicit guaranty the input points are at the predicted shape function level set from a single forward pass. We hypothesise that enforcing this constraint under well chosen regularizations could potentially lead to improvements at test time and a reduction of the generalization issues mentioned above.
We want to formulate an efficient strategy to improve on these aspects of generalizable shape networks at test time, i.e. reconstructing shapes by dynamically adjusting the network to the input point-cloud during testing. We note that we follow such a strategy as training large models from scratch can be resource intensive, while adaptation can be less costly and has been shown to yield considerable improvements in e.g. domain adaptation and transfer learning literature. One possible solution [69] (SAC) is to finetune the network weights at test time, using the knowledge that input point cloud samples are on the desired surface as supervision. However, we find this strategy to be unstable as it can overfit on these samples. As shown throughout the results (Section 4), it even exacerbates the performance of its initial baseline network in many cases. This instability highlights the necessity for regularization in this adaptation process. We need to balance the need to maintain the expressive capacity of the learned functions while ensuring stability and robustness. To this end we devise three important design choices:
- We restrain the hypothesis space of the adapted shape functions to be a Reproducing Kernel Hilbert Space (RKHS). By the Representative Theorem, the minimizer of our regularized empirical risk minimization (ERM) problem (Equation 7) emerges naturally as the solution of a Kernel Ridge Regression (KRR) problem.
- By using a Gaussian kernel in this context, we benefit from the universality properties of the associated (RKHS) i.e. a hypothesis space rich enough to approximate any continuous function arbitrarily well. Thus, we avoid the difficulties arising from optimizing a large number of neural network parameters, as in SAC, while maintaining the expressive capacity necessary for effective shape modeling.
- We propose a strategy to solve our regularized ERM problem in a RHKS space adapted to the shape we would like to recover. This is possible thanks to the unique correspondence between RKHS and kernels. Hence, instead of relying on handmade fixed kernels, we learn the KRR hyperparameters using a loss function that avoids overfitting on the data term (Equation 11).
In this regard, approximate KRR solvers based on Nyström samples (Nyström KRR) [48] can be computed efficiently and allow scaling in point cloud size. As shown in our ablation studies, defining this kernel in euclidean space seems insufficient to constrain the fitting problem. Hence we define it in the feature space of the pretrained convolutional shape network. To define the fitting data, we use the input point cloud with their inherent label: being samples from the surface by default. We also augment the fitting data with additional samples paired with their shape network predictions as pseudo-labels. This augmentation ensures the fitting problem is well defined. We note that through this strategy, knowledge is being transferred from the network to our kernel regression through both features and pseudo-labels.
Until recently, exiting methods for KRR hyperparameter tuning, e.g. cross-validation or grid search, placed strong limits on the flexibility and number of hyperparameters that can be tuned. The work of Meanti et al. [47] proposed lately a automated gradient based tuning of the hyperparameters of the KRR by minimizing a loss that prevents overfitting on the fitting data. This strategy allows the optimization of both the kernel hyperprameters and the Nyström samples, and proves beneficial in our context, i.e. fitting a shape function in feature space, as compared to baselines with fixed Nyström samples and/or kernel hyperparameters (See ablation in section 4.7). A visual summary of our method can be found in Figure 1.
To test our idea we devise experiments on real and synthetic data targeting generalization issues. Our approach improves consistently on the baseline networks, and outperforms other state-of-the-art methods and test time network finetuning based strategies. We also show that our approach can be applied successfully to more than one network. Our ablation studies showcase also the importance of all the various components of our method.
We review in this section previous work we deemed most relevant to our problem and contribution.
Shape Representations in Deep Learning Shapes can be represented within deep learning either intrinsically or extrinsically. Intrinsic representations are a discrimination of the shape itself. When done explicitly, using e.g. tetrahedral or polygonal meshes [73, 32] or point clouds [22], the output topology in predefined thus bounding the variability of the shapes that can be generated. Among other forms of intrinsic representations, 2D patches [26, 79, 20] can prompt discontinuities, whilst the simplicity of shape primitives such as cuboids [72, 86], planes [39] and Gaussians [24] limits their expressiveness. Differently, extrinsic shape representations model the entire space containing the scene or object of interest. Voxel grids [82, 81] are the most popular one being the direct extension of 2D pixels to 3D domain. However, their capacity is limited by their cubic resolution memory cost. Sparse representations such as octrees [60, 70, 75] can alleviate this issue to some extent.
Implicit Neural Shape Representations Implicit neural representations (INRs) emerged recently as a major medium for modelling extrinsic shape and radiance fields (e.g. [51, 83, 74, 30, 11]). They overcome many of the limitations of the aforementioned classical representations thanks to their ability to represent shapes with arbitrary topologies at virtually infinite resolution. They are usually parameterised with MLPs mapping spatial locations or features to e.g. occupancy [50], signed [55] or unsigned [16, 85] distances relative to the target shape. The level-set of the inferred field from these MLPs can be rendered through ray marching [28], or tessellated into an explicit shape using e.g. Marching Cubes [43]. Another noteworthy branch of work builds hybrid implicit/explicit representations [54, 14, 19, 84] based mostly on differentiable space partitioning. In order to represent collections of shape simultaneously, implicit neural models require conditioning mechanisms. These include feature and latent code concatenation, batch normalization, hypernetworks [65, 67, 66, 76, 13] and gradient-based meta-learning [52, 64]. Concatenation based conditioning was first implemented using single global latent codes [50, 15, 55], and further improved with the use of local features [53, 35, 23, 71, 68, 57, 17, 31, 21].
Reconstruction from Point Cloud Classical approaches include combinatorical ones where the shape is defined through an input point cloud based space partitioning, through e.g. alpha shapes [5] Voronoi diagrams [1] or triangulation [9, 41, 59]. On the other hand, the input samples can be used to define an implicit function whose zero level set represents the target shape, using global smoothing priors [78, 36, 80] e.g. radial basis function [8] and Gaussian kernel fitting [61], local smoothing priors such as moving least squares [49, 27, 34, 42], or by solving a boundary conditioned Poisson equation [33]. The recent literature proposes to parameterise these implicit functions with deep neural networks and learn their parameters with gradient descent, either in a supervised or unsupervised manner.
Unsupervised Implicit Neural Reconstruction A neural network is typically fitted to the input point cloud without extra information. Regularizations can improve the convergence such as the spatial gradient constraint based on the Eikonal equation introduced by Gropp et al. [25], a spatial divergence constraint as in [4], Lipschitz regularization on the network [40]. Atzmon et al. learns an SDF from unsigned distances [2], and further supervises the spatial gradient of the function with normals [3]. Ma et al. [44] expresses the nearest point on the surface as a function of the neural signed distance and its gradient. They also leverage self-supervised local priors to deal with very sparse inputs [45] and improve generalization [46]. All of the aforementioned work benefits from efficient gradient computation through back-propagation in the neural network. Periodic activations were introduced in [65]. Lipman [38] learns a function that converges to occupancy while its log transform converges to a distance function. [80] learns infinitely wide shallow MLPs as random feature kernels using points and their normals. Most of the aforementioned methods present failures under sparse and noisy input due to the lack of supervision and data priors. Differently, we propose here to combine a data prior based method and self-supervised learning.
Supervised Implicit Neural Reconstruction Supervised methods assume a labeled training data corpus commonly in the form of dense samples with ground truth shape information. Auto-decoding methods [35, 55, 71, 31, 10] require test time optimization to be fitted to a new point cloud, which can take up to several seconds. Encoder-decoder based methods enable fast feed forward inference. Introduced first in this respect, Pooling-based set encoders [50, 15, 23] such as PointNet [58] have been shown to underfit the context. Convolutional encoders yield state-of-the-art performances. They use local features either defined in explicit volumes and planes [57, 17, 37, 56] or solely at the input points [7]. Ouasfi et al. [53] proposed concurrently the first convolution-free fast feed forward generalizable model. Peng et al. [56] proposed a differentiable Poisson solving layer that converts predicted normals into an indicator function grid efficiently. However, it is limited to small scenes
Figure 1: Overview. Our method predicts an implicit shape function from a noisy unoriented input point cloud. We combine a cross-shape deep prior (Pretrained generalizable occupancy network) and an intra-shape adaptive Nyström Kernel Ridge Regression (NKRR)  $g$  at test time. The latter learns to map network features of the input points and additional points to the level-set and network generated pseudo-labels respectively. The NKRR hyperparameters  $(\mathbf{Z},\boldsymbol{\theta})$  are adjusted to the current shape.
due to the cubic memory requirement in grid resolution. Most of these methods still suffer from generalization issues. Concurrently, the work in [78, 29] proposes to build generalizable models where the implicit decoder is a kernel regression. Differently, we advocate a strategy to improve the generalization of any preexisting pretrained reconstruction network.
Our input is a noisy unoriented point cloud  $\mathbf{P} \subset \mathbb{R}^{3 \times N_p}$ . Our objective is to recover a 3D reconstruction, i.e. the shape surface  $\mathcal{S}$  that best explains this observation, where the input point cloud elements approximate noisy samples from  $\mathcal{S}$ .
To achieve this, we model an implicit function  $f$  that predicts occupancy values relative to a target shape  $S$  at any queried euclidean space location  $q \in \mathbb{R}^3$ , given the input point cloud  $\mathbf{P}$ , i.e.  $f(\mathbf{P}, q) = 1$  if  $q$  is inside the shape, and 0 otherwise. The inferred shape  $\hat{S}$  can then be obtained as a level set of the occupancy field inferred through  $f$ :
$$
\hat {\mathcal {S}} = \{q \in \mathbb {R} ^ {3} \mid f (\mathbf {P}, q) = 0. 5 \}. \tag {1}
$$
In practice, an explicit triangle mesh for  $\hat{S}$  is extracted using the Marching Cubes [43] algorithm.
3.1 Feedforward Generalizable Model
The staple back-bone models for feedforward generalizable implicit shape reconstruction from point cloud (e.g. [57, 17, 7]) have typically a two-stage architecture. First, a deep convolutional network (usually a U-Net) builds features in  $\mathbb{R}^{N_F}$  from the input point cloud:
$$
\mathbf {F} = \Phi (\mathbf {P}). \tag {2}
$$
Features  $\mathbf{F}$  can be extrinsic or intrinsic. For instance in [57, 17], the ConvNet builds an extrinsic 2D or 3D explicit feature grid representing space around the shape. More recently, Boulch et al. [7] argued against this strategy, contending that voxel centers may be far from the input point cloud locations, and are sampled uniformly while they should be more focused near the surface of interest. They hence define features intrinsically, i.e. only at the input point cloud locations. In both strategies, the ConvNet offers transnational equivariance and generalization ability. It is worth noting that it is also naturally endowed with a locality mechanism, which is crucial for performance with fast inference times, as only a single forward pass in the encoder is needed to build the feature space. Conversely, other generalizable models lacking locality in their encoders, such as the work in [21], enforce locality through local patch inputs. They hence require as many encoder forward passes as there are query points. This naturally results in significantly increased inference times compared the aforementioned competition that we build on.
Given a 3D query point  $q$ , an implicit decoder, typically an MLP, maps the feature to the occupancy value:
$$
\begin{array}{l} \operatorname {o c c} (q) = \sigma (\Psi (\mathbf {F} (q))) (3) \\ = \sigma (l (q)), (4) \\ \end{array}
$$
where  $\sigma(.)$  represents the Sigmoid activation and  $l(.)$  a logit function. In the case of extrinsic feature networks [57, 17, 56], feature  $\mathbf{F}(q)$  is obtained by linear interpolation (trilinear in the case of 3D feature grids, and bilinear for 2D grids). For intrinsic feature networks [7], the feature is pooled order-invariantly from the nearest points of the input point cloud. Without loss of generality we build in this work on the model in [7], as it has been shown to achieve state-of-the-art performances on several benchmarks.
3.2 Kernel Ridge Regression as a Shape Function in Feature Space
Existing pretrained generalizable feedforward models for reconstruction from point cloud suffer from generalization limitations. As they are typically trained on synthetic shape datasets (e.g. ShapeNet [12]) with full supervision, using a predefined point cloud size and simulated noise, their performance reduces when tested on point clouds that are different from the training ones regarding properties such as density, and whether they are real or simulated. Such shortcomings of an instance of these models are showcased in Tables 7 and 5. We design henceforth an efficient mechanism that can fill this performance gap, while benefiting from the available shape labels at test time: The fact that the input point cloud is made of samples that belong to the surface we wish to reconstruct.
We consider the following training data samples:
$$
\{(x _ {i}, y _ {i}) \} _ {i = 1} ^ {n} = \{(\mathbf {F} (p _ {i}), 0) \} _ {i = 1} ^ {N _ {p}} \cup \{(\mathbf {F} (\widetilde {p} _ {i}), l (\widetilde {p} _ {i})) \} _ {i = 1} ^ {N _ {p}}, \tag {5}
$$
where  $(x_{i},y_{i})\in \mathbb{R}^{N_{F}}\times \mathbb{R}$  and  $n = 2\times N_p$ .  $\{p_i\}$  are the samples of the input point cloud  $\mathbf{P}$ .  $\{\widetilde{p}_i\}$  are additional samples around this point cloud, that can be automatically generated following e.g. the strategy in [44]. Features  $\mathbf{F}(.)$  and occupancy logs  $l(.)$  are obtained from the pretrained convolutional occupancy model (See Equations 2 and 4). We note that these data samples are corrupted by a combination of observation noise in the input point cloud (points  $p_i$  not being exactly at the surface) and prediction errors of the initial model (pseudo-labels  $l(\widetilde{p}_i)$  being inaccurate).
We approximate the mapping from deep shape features  $x_{i}$  to occupancy logits  $y_{i}$  using a non-linear non-parametric regression  $g(. )$  to provide spatial regularization while favouring flexibility, i.e. without making strong assumptions on the model. In particular, we use Kernel Ridge Regression (KRR), where the hypothesis space of the mapping is a reproducing kernel Hilbert space  $\mathcal{H}$ . Associated to this space is a kernel function  $k_{\theta}: \mathbb{R}^{N_F} \times \mathbb{R}^{N_F} \to \mathbb{R}$ , that depends on hyperparameters  $\theta$ . To ensure the minimisation of the square error of the mapping over the training samples is well defined, a regularization is typically needed, with a weight  $\lambda$ :
$$
\hat {g} _ {\boldsymbol {\theta}} = \underset {g \in \mathcal {H}} {\arg \min } \| g (\mathbf {X}) - \mathbf {Y} \| ^ {2} + \lambda \| g \| _ {\mathcal {H}} ^ {2}, \tag {6}
$$
$$
\text {w h e r e} \quad \mathbf {X} = \left\{x _ {i} \right\} _ {i = 1} ^ {n}, \quad \mathbf {Y} = \left\{y _ {i} \right\} _ {i = 1} ^ {n}. \tag {7}
$$
The unique solution to this problem is expensive to compute. For efficiency we follow the approximation to KRR that considers a lower subspace of dimension  $m << n$  [77]. This allows our method to scale to large input point clouds. The approximation uses  $m$  inducing features  $\mathbf{Z} = \{z_{j}\}_{j=1}^{m}$ , also known as Nyström Samples, where  $z_{j} \in \mathbb{R}^{N_{F}}$ .
Finally, the unique solution to the minimization in Equation 6 writes [48]:
$$
\hat {g} _ {\boldsymbol {\theta}, \mathbf {Z}} = \sum_ {j = 1} ^ {m} \beta_ {j} k _ {\boldsymbol {\theta}} (\cdot , z _ {j}), \tag {8}
$$
$$
\text {w h e r e} \quad \boldsymbol {\beta} = \left(\mathbf {K} _ {n m} ^ {\top} \mathbf {K} _ {n m} + \lambda n \mathbf {K} _ {m m}\right) ^ {- 1} \mathbf {K} _ {n m} ^ {\top} \mathbf {Y}. \tag {9}
$$
Here,  $(\mathbf{K}_{nm})_{i,j} = k_{\pmb{\theta}}(x_i,z_j)$  and  $(\mathbf{K}_{mm})_{i,j} = k_{\pmb{\theta}}(z_i,z_j)$ , where  $i\in [[1,n]]$  and  $j\in [[1,m]]$ . This solution can be computed efficiently through e.g. the Falcon solver [48].
3.3 Learning the Shape Function Hyperparameters
Next, based on [47], we propose to automatically learn the hyperparameters of our Nyström KRR, namely the kernel hyperparameters  $\theta$  and the Nyström samples  $\mathbf{Z}$ , using gradient descent. This shape specific tuning allows for a larger hyperparameter space and using more expressive kernels. The optimization consists in minimizing an upper bound  $\mathcal{L}^{\mathrm{tune}}$  of the ideal objective, i.e. the expectation over all possible data points of the squared error of the model. This allows to avoid overfitting on the training set:
$$
\mathcal {L} ^ {\text {t u n e}} = \frac {2}{n} \operatorname {T r} \left(\left(\widetilde {\mathbf {K}} + n \lambda \mathbf {K}\right) ^ {- 1} \widetilde {\mathbf {K}}\right) + \frac {2}{n \lambda} \operatorname {T r} (\mathbf {K} - \widetilde {\mathbf {K}}) \mathcal {L} ^ {\text {d a t a}} \left(\hat {g} _ {\boldsymbol {\theta}, \mathbf {Z}}\right) + 2 \mathcal {L} ^ {\text {d a t a}} \left(\hat {g} _ {\boldsymbol {\theta}, \mathbf {Z}}\right), \tag {10}
$$
$$
\text {w h e r e} \quad \mathcal {L} ^ {\mathrm {d a t a}} \left(\hat {g} _ {\boldsymbol {\theta}, \mathbf {Z}}\right) = \frac {1}{n} \| \hat {g} _ {\boldsymbol {\theta}, \mathbf {Z}} (\mathbf {X}) - \mathbf {Y} \| ^ {2} + \lambda \| \hat {g} _ {\boldsymbol {\theta}, \mathbf {Z}} \| _ {\mathcal {H}} ^ {2}. \tag {11}
$$
$\widetilde{\mathbf{K}} = \mathbf{K}_{nm}\mathbf{K}_{mm}^{\dagger}\mathbf{K}_{nm}^{\top}$  and  $\dagger$  symbolizes the Moore-Penrose matrix inverse. The upper bound is made, in this order, of the Nyström penalty attempting to make the Nyström kernel  $\widetilde{\mathbf{K}}$  closer to the complete kernel  $\mathbf{K}$ , a second term penalizing overly complex kernels, and finally a data fit term. We note that the authors in [47] derived this loss by replacing expectations with their empirical counterparts. They also propose to use the Hutchinson approximation to compute matrix traces for efficiency.
After  $N$  gradient descent steps (as illustrated in Algorithm 1), we select hyperparameters  $\theta^{*}$  and  $\mathbf{Z}^*$  that minimize the squared error of the model over the training data following [47]. For a given query point q, our final implicit occupancy function can be expressed as the sigmoid activated optimal Nyström KRR in feature space:
$$
f (\mathbf {P}, q) = \sigma \left(\hat {g} _ {\boldsymbol {\theta} ^ {*}, \mathbf {Z} ^ {*}} (\mathbf {F} (q))\right) \tag {12}
$$