Robustifying Generalizable Implicit Shape Networks with a Tunable Non-Parametric Model
Authors: Amine Ouasfi
Adnane Boukhayma
Venue: NeurIPS 2023 poster (2023)
Keywords: implicit neural representations
3D reconstruction from unoriented point could
kernel ridge regression

Abstract: Feedforward generalizable models for implicit shape reconstruction from unoriented point cloud present multiple advantages, including high performance and inference speed. However, they still suffer from generalization issues, ranging from underfitting the input point cloud, to misrepresenting samples outside of the training data distribution, or with toplogies unseen at training.  We propose here an efficient mechanism to remedy some of these limitations at test time. We combine the inter-shape data prior of the network with an intra-shape regularization prior of a Nyström Kernel Ridge Regression, that we further adapt by fitting its hyperprameters to the current shape. The resulting shape function defined in a shape specific Reproducing Kernel Hilbert Space benefits from desirable stability and efficiency properties and grants a shape adaptive expressiveness-robustness trade-off. We demonstrate the improvement obtained through our method  with respect to baselines and the state-of-the-art using synthetic and real data.

Robustifying Generalizable Implicit Shape Networks with a Tunable Non-Parametric Model
Adnane Boukhayma Inria, Univ. Rennes, CNRS, IRISA, M2S, France
Amine Ouasfi
Abstract
Feedforward generalizable models for implicit shape reconstruction from unori- ented point cloud present multiple advantages, including high performance and inference speed. However, they still suffer from generalization issues, ranging from underfitting the input point cloud, to misrepresenting samples outside of the training data distribution, or with toplogies unseen at training. We propose here an efficient mechanism remedy some of these limitations at test time. We combine the inter-shape data prior of the network with an intra-shape regularization prior of Nyström Kernel Ridge Regression, that we further adapt by fitting its hyperprameters to the current shape. The resulting shape function defined in a shape specific Reproducing Kernel Hilbert Space benefits from desirable stability and efficiency properties and grants a shape adaptive pepressienees-robustnne trade-off. We demonstrate the improvement obtained through our method with respect to baselines and the state-of-the-ar using synthetic and real data.
Introduction
Enabling machines to understand and navigate 3D a major challenge. Instances of this ability include downstream tasks in computer vision and graphics, such as shape reconstruction from noisy, incomplete and relatively sparse point clouds. Building full shapes from point clouds is all the more an important problem in account of the ubiquity of this light, albeit incomplete, 3D shape representation, whether it is acquired from the increasingly democratized consumer grade and industrial depth sensors e.g. obtained from photogrammetry (e.g. Structure From Motion, Multi- View Stereo [62, 63]). Whilst Classical optimization based approaches such as Poisson Reconstruction [33] and Moving least squares [27] can deliver mostly good reconstructions for all intents and purposes, they require still dense clean point sets with reliable normal estimations. More recently, deep learning based alternatives have been shown offer faster and more robust predictions especially for noisy and sparse inputs, without requiring normal information.
Within this class of methods, state-of-the-art feed forward (test time optimization-freet generalizable models are based on implicit neural shape representations, where explicit meshes can be extracted at test time using Marching Cubes [43]. While they offer many desirable properties, such all-round good performance and fast inference (within tens of milliseconds on a standard GPU), they still suffer from limitations, some of which come across upon examining the state-of-the--e the--rt art method [7] for instance:
Limited OOD generalization As these models typically trained with full 3D supervision, training commonly uses synthetic data such as the ShapeNet [12] dataset, large real 3D supervision being famously hard to come by. This results in performance drop for real test data outside the training distribution. Comparing Tables 2 and 6, the performance of [7] drops by roughly fourfold going from testing on synthetic data to real scans.
Limited input size generalization As illustrated in table 7, when presented with testing point clouds that larger (e.g. 10k) than the training size (3k), the performance does not scale and even deteriorates compared to testing on the training size.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Underfitting the input There is no explicit guaranty the input points are at the predicted shape function level set from a single forward pass. We hypothesise that enforcing this constraint under well chosen regularizations could potentially lead to improvements at test time and a reduction of the generalization issues mentioned above.
We want to formulate an efficient strategy to improve on these aspects of generalizable shape networks test time, i.e. reconstructing shapes by dynamically adjusting the network the input point-cloud during testing. We note that we follow such a strategy as training large models from scratch can be resource intensive, while adaptation can be less costly and has been shown to yield considerable improvements in e.g. domain adaptation and transfer learning literature. One possible solution [69] (SAC) is to finetune the network weights test time, using the knowledge that input point cloud samples are on the desired surface as supervision. However, we find this strategy to be unstable as it can overfit on these samples. As shown throughout the results (Section 4), it even exacerbates the performance of its initial baseline network in many cases. This instability highlights the necessity for regularization in this adaptation process. We need to balance the need to maintain the expressive capacity of the learned functions while ensuring stability and robustness. To this end we devise three important design choices:
We restrain the hypothesis space of the adapted shape functions to be Reproducing Kernel Hilbert Space (RKHS). By the Representer Theorem, the minimizer of our regularized empirical risk minimization (ERM) problem (Equation 7) emerges naturally the solution of Kerel Ridge Regression (KRR) problem.
By using a Gaussian kernel in this context, we benefit from the universality properties of the associated (RKHS) ie. a hypothesis space rich enough to approximate any continuous function arbitrarily well. Thus, we avoid the difficulties arising from optimizing large number of neural network parameters, as in SAC, while maintaining the expressive capacity necessary for effective shape modeling.
We propose a strategy to solve our regularized ERM problem in RHKS space adapted to the shape we would like recover. This is possible thanks the unique correspondence between RKHS and kernels. Hence, instead of relying on handmade fixed kemels, we learn the KRR hyperparameters using a loss function that avoids overfitting on the data term (Equation 11).
In this regard, approximate KRR solvers based on Nyström samples (Nyström KRR) [48] can be computed efficiently and allow scaling in point cloud size. As shown in our ablation studies, defining this kernel in euclidean space seems insufficient to constrain the fitting problem. Hence we define it in the feature space of the pretrained convolutional shape network. To define the fitting data, use the input point cloud with their inherent label: being samples from the surface by default. We also augment the fitting data with additional samples paired with their shape network predictions as pseudo-labels. This augmentation ensures the fitting problem is well defined. We note that through this strategy, knowledge is being transferred from the network to our kernel regression through both features and pseudo-labels.
Until recently, exiting methods for KRR hyperparameter tuning, e.g. cross- vooss-valiattinn grid search, placed strong limits on the flexibility and number of hyperparameters that can be tuned. The work of Meanti al. [47] proposed lately automated gradient based tuning of the hyperparameters of the KRR by minimizing a loss that prevents overfitting on the fitting data. This strategy allows the optimization of both the kernel hyperprameters and the Nyström samples, and proves beneficial in our context, i.e. fitting a shape function in feature space, as compared to baselines with fixed Nyström samples and/or kernel hyperparameters (See ablation in section 4.7). A visual summary of our method can be found in Figure 1.
To test our idea we devise experiments real and synthetic data targeting generalization issues. Our approach improves consistently on the baseline networks, and outperforms other state-of-the-a methods and test time network finetuning based strategies. We also show that our approach can be applied successfully to more than one network. Our ablation studies showcase also the importance of all the various components of our method.
2
Related Work
We review in this section previous work we deemed most relevant to our problem and contribution.

---

Shape Representations in Deep Learning Shapes can be represented within deep learning either intrinsically or extrinsically. Intrinsic representations are a discrimination of the shape itself. When done explicitly, using e.g. tetrahedral or polygonal meshes [73, 32] or point clouds [22]. the output topology in predefined thus bounding the variability of the shapes that can be generated. Among other forms of intrinsic representations, 2D patches [26, 79, 20] can prompt discontinuities, whilst the simplicity of shape primitives such as cuboids [72, 86], planes [39] and Gaussians [24] limits their expressiveness. Differently, extrinsic shape representations model the entire space containing the scene or object of interest. Voxel grids [82, 81] are the most popular one being the direct extension of 2D pixels to 3D domain. However, their capacity is limited by their cubic resolution memory cost. Sparse representations such as octrees [60, 70, 75] can alleviate this issue to some extent.
Implicit Neural Shape Representations Implicit neural representations (INRs) emerged recently as a major medium for modelling extrinsic shape and radiance fields (e.g. [51, 83, 74, 30, 11]). They overcome many of the limitations of the aforementioned classical representations thanks to their ability to represent shapes with arbitrary topologies at virtually infinite resolution. They are usually parameterised with MLPs mapping spatial locations or features to e.g. occupancy [50], signed [55] or unsigned [16, 85] distances relative to the target shape. The level-set of the inferred field from these MLPs can be rendered through ray marching [28], or tessellated into an explicit shape using e.g. Marching Cubes [43]. Another noteworthy branch of work builds hybrid implicit/explict representations [54, 14, 19, 84] based mostly on differentiable space partitioning. In order to represent collections of shape simultancously, implicit neural models require conditioning mechanisms. These include feature and latent code concatenation, batch normalization, hypernetworss [65, 67, 66, 76, 13] and pradient-based meta-learning [52, 64]. Concatenation based conditioning was first implemented using single global latent codes [50, 15, 55], and further improved with the use of local features [53, 35, 23, 71, 68, 57, 17, 31, 21].
Reconstruction from Point Cloud Classical approaches include combinatorical ones where the shape defined through an input point cloud based space partitioning, through e.g. alpha shapes [5] Voronoi diagrams [1] or triangulation [9, 41, 591. On the other hand, the input samples can be used to define an implicit function whose zero level set represents the target shape, using global smoothing
priors [78, 36, 80] e.g. radial basis function [8] and Gaussian kernel fitting [61], local smoothing
priors such as moving least squares [49, 27, 34, 42], or by solving a boundary conditioned Poisson equation [33]. The recent literature proposes to parameterise these implicit functions with deep neural networks and learn their parameters with gradient descent, either in a supervised or unsupervised manner.
Unsupervised Implicit Neural Reconstruction A neural network is typically fitted to the input point cloud without extra information. Regularizations can improve the convergence such as the spatial gradient constraint based on the Eikonal equation introduced by Gropp et al. [25], a spatial divergence constraint as in [4], Lipschitz regularization on the network [40]. Atzmon et al. learns an SDF from unsigned distances [2], and further supervises the spatial gradient of the function with normals [3]. Ma et al. [44] expresses the nearest point on the surface as a function of the neural signed distance and its gradient. They also leverage self-supervised local priors to deal with very sparse inputs [45] and improve generalization [46]. All of the aforementioned work benefits from efficient gradient computation through back pocp-propngation in the neural network. Periodic activations were introduced in [65]. Lipman [38] learns function that converges to occupancy while its log transform converges to distance function. [80] learns infinitely wide shallow MLPs as random feature kernels using points and their normals. Most of the aforementioned methods present failures under sparse and noisy input due to the lack of supervision and data priors. Differently, we propose here to combine a data prior based method and self-supervised learning.
Supervised Implicit Neural Reconstruction Supervised methods assume a labeled training data corpus commonly in the form of dense samples with ground truth shape information. Auto-decoding methods [35, 55, 71, 31, 10] require test time optimization be fitted new point cloud, which can take up to several seconds. Encoder-decoder based methods enable fast feed forward inference. Introduced first in this respect, Pooling- -bsee set encoders [50, 15, 23] such as PointNet [58] have been shown to underfit the context. Convolutional encoders yield state-of -ft-----thhhe-eeee performances. They use local features either defined in explicit volumes and planes [57, 17, 37, 56] or solely at the input points [7]. Ouasfi el al. [53] proposed concurrently the first convolution-fres fast feed forward generalizable model. Peng el al. [56] proposed a differentiable Poisson solving layer that converts predicted normals into an indicator function grid efficiently. However, it is limited to small scenes
3

---

Pretrained
V
Generalizable
[PP,) о c (F)))
Octupprtiy
$ {g S(P. g) = 0.5)
F
solve
90.2
Cunne
F(P.). (paa)
initiallzze
Nystrom
samples
Z
kemel
θ
S
update
Figure 1: Overview. Our method predicts implicit shape function from noisy unoriented input point cloud. We combine a cross-shape deep prior (Pretrained generalizable occupancy network) and an intra-shape adaptive Nyström Kernel Ridge Regression (NKRR) g at test time. The latter learns to map network features of the imput points and additional points to the level-set and network generated pseudo-labels respectively. The NKRR hyperparameters (Z,0) adjusted to the current shape.
due to the cubic memory requirement grid resolution. Most of these methods still suffer from generalization issues. Concurrently, the work in [78, 29] proposes to build generalizable models where the implicit decoder a kernel regression. Differently, we advocate a strategy to improve the generalization of any preexisting pretrained reconstruction network.
3 Method
Our input a noisy unoriented point cloud P RR3xN,, Our objective to recover 3D reconstruc- tion, i.e. the shape surface S that best explains this observation, where the input point cloud elements approximate noisy samples from S.
To achieve this, we model an implicit function f that predicts occupancy values relative to target shape any queried euclidean space location q E R&, given the input point cloud P, i.e. f(P.q) 1 if q is inside the shape, and 0 otherwise. The inferred shape S can then be obtained as a level set of the occupancy field inferred through f:
S = R (PP.)) = 0.5}.
(1)
In practice, an explicit triangle mesh for S is extracted using the Marching Cubes [43] algorithm.
3.1 Feedforward Generalizable Model
The staple back- bone models for feedforward generalizable implicit shape reconstruction from point cloud (e.g. [57, 17, 7J) have typically two-stage architecture. First, deep convolutional network (usually a U-Net) builds features in N from the input point cloud:
F == (PP)
(2)
Features F can be extrinsic intrinsic. For instance in [57, 17], the ConvNet builds extrinsic 2D or 3D explicit feature grid representing space around the shape. More recently, Boulch et al. [7] argued against this strategy, contending that voxel centers may be far from the input point cloud locations, and are sampled uniformly while they should be more focused near the surface of interest. They hence define features intrinsically, i.e. only at the input point cloud locations. the ConvNet offers transnational equivariance and generalization ability. It is worth noting that it also naturally endowed with a locality mechanism, which crucial for performance with fast inference times, as only single forward pass in the encoder is needed to build the feature space. Conversely, other generalizable models lacking locality their encoders, such as the work [21], enforce locality through local patch inputs. They hence require as many encoder forward passes as there are query points. This naturally results in significantly increased inference times compared the the aforementioned competition that we build on.
both strategies,

---

Given 3D query point q. an implicit decoder, typically an MLP, maps the feature to the occupancy value:
occ - (V((F())
= =
where (.) represents the Sigmoid activation and ((-) logit function. In the case of extrinsic feature networks [57, 17, 56], feature F ((g) is obtained by linear interpolation (trilinear in the case of 3D feature grids, and bilinear for 2D grids). For intrinsic feature networks [7], the feature is pooled order-invariantly from the nearest points of the input point cloud. Without loss of generality build in this work on the model in [7], as has been shown to achieve state-of-the-art performances on several benchmarks.
3.2 Kernel Ridge Regression as a Shape Function in Feature Space
Existing pretrained generalizable feedforward models for reconstruction from point cloud suffer from generalization limitations. As they typically trained on synthetic shape datasets (e.g. ShapeNet (12]) with full supervision, using predefined point cloud size and simulated noise, their performance reduces when tested on point clouds that are different from the training ones regarding properties such as density, and whether they are real or simulated. Such shortcomings of an instance of these models are showcased in Tables and 5. We design henceforth an efficient mechanism that can fill this performance gap, while benefiting from the available shape labels at test time: The fact that the input point cloud is made of samples that belong to the surface we wish reconstruct.
We consider the following training data samples:
(5) where (xz, V/) E R x R and n = 2 x Np. {p:} are the samples of the input point cloud P. {p:} are additional samples around this point cloud, that can be automatically generated following egg. the strategy in [44]. Features F()) and occupancy logits ((.) are obtained from the pretrained convolutional occupancy model (See Equations 2 and 4). We note that these data samples corrupted by a combination of observation noise in the input point cloud (points pt not being exactly the surface) and prediction errors of the initial model (pseudo-labels ((x) being inaccurate).
{1 y0 (F(P.). ((FO))
We approximate the mapping from deep shape features x; to occupancy logits y using a non-lincar conpparametrc regression g(.) to provide spatial regularization while favouring flexibility, i.e. without making strong assumptions on the model. In particular, we use Kernel Ridge Regression (KRR), where the hypothesis space of the mapping is reproducing kernel Hilbert space H. Associated to this space is a kernel function kg RN x R Nr + R, that depends on hyperparameters O. To ensure the minimisation of the square error of the mapping over the training samples is well defined, a regularization is typically needed, with a weight X:
go = arg minllg(X) - 111 + Alggl
(6)
9E7
where
x (zi
Y = (3))
The unique solution to this problem is expensive compute. For efficiency we follow the approxima- tion to KRR that considers a lower subspace of dimension m << n [77]. This allows our method to scale to large input point clouds. The approximation uses m inducing features Z {2,} also known as Nyström Samples, where zj E Nr.
Finally, the unique solution to the minimization in Equation 6 writes [48]:
gØ.z Σ 3,ko (3)zz),
(8)
where B [ Kum + AnKmm
KT Y.
(9)
Here, (Kam)s, ke (I 23) and (Kmm)a, ke (2), 25), where E [[1,n] and j E [[1, m]. This solution can be computed efficiently through e.g. the Falcon solver [48].
5

---

3.3 Learning the Shape Function Hyperparameters
Next, based on [47], we propose to automatically learn the hyperparameters of our Nyström KRR, namely the kernel hyperparameters 0 and the Nyström samples Z, using gradient descent. This shape specific tuning allows for a larger hyperparameter space and using more expressive kernels. The optimization consists in minimizing upper bound Lume of the ideal objective, i.e. the expectation over all possible data points of the squared error of the model. This allows to avoid overfitting on the training set:
2
2 K
Cune
Tr (K+
(
Tr(K L (0o,z) 2. (
(90,z);
(10)
nd
1
where
Canta (āo .))
1- 1190.22 (X) - YII + X1100
(11)
mm KT am and symbolizes the Moore-Penrose matrix inverse. The upper bound is made, in this order, of he Nyström penalty attempting to make the Nyström kernel K closer to the complete kernel K, second term penalizing overly complex kernels, and finally data fit term. We note that the authors in [47] derived this loss by replacing expectations with their empirical counterparts. They also propose to use the Hutchinson approximation to compute matrix traces for efficiency.
K K- K K
Aftre N gradient descent steps (as illustrated in Algorithm 1), select hyperparameters 0" and Z that minimize the squared error of the model over the training data following [47]. For a given query point q, our final implicit occupancy function can be expressed as the sigmoid activated optimal Nyström KRR in feature space:
f(P.g) = 0 z (F()))
(12)
Algorithm 1 The training procedure of our method.
Point cloud P. pretrained occupancy network (V, b), learning Output: optimal kernel hyperparameters 6" and Nyström samples Z"
Input:
P upsample(P)
X P))
Y 10, V o ((P)
initialize Z as random features from initialize
for N times
compute ge. from X, Y (Equations 8 and 9) compute L for ğo. X and Y (Equations 10 and 11)
(0, Z) (0, Z) aV L
end for
4 Results
In this section, we evaluate our method using real and synthetic data both quantitatively and qualita- tively. As our method builds Poco [7], compare our results this baseline. We also compare to other state-of-the-art convolutional occupancy networks, such as Conv [57] and SAP [56]. We compare the strategy based test-time finetuning of convolutional occupancy network (Conv [57]) ) in SAC [69]. Additionally, we compare to the strategy based on fitting an MLP to the point cloud in NP [44]. We compare also classical learning-fit Poisson reconstruction (SPSR [33]) where input point normals are estimated with PCA. For state-of-the-an convolutional occupancy networks Poco, Conv and SAP, we use their official publicly available pretrained models as trained on the same training split of ShapeNet [12] with 3k sized point clouds and noise of variance 0.005. We use the publicly available official implementations of SAP and NP. We present also an ablative analysis of the components of our method in Section 4.7. We provide additional results in the supplementary material. We note that from a computational stand point, for 10k sized input point cloud, our method takes roughly 10 seconds to converge on a NVIDIA RTX A6000 GPU. The other test-time tuning based alternatives we compare to here, namely SAC and NP take roughly and minutes respectively.

---

4.1 Implementation Details
We run all the experiments for 100 epochs using Adam optimizer with learning rate 0.1. We use the Falcon Library to implement the tuning of Nyström KRR in feature space. Regarding the input size and feature dimensions, We experimented with Np 10k (also 500 and 3k in ablation) and Np = 32 and m = 500 inducing points.
4.2 Metrics
Following seminal work, we evaluate our method and the competition w.r.t. the ground truth using standard metrics for the 3D reconstruction task. Namely, the volumetric Intersection over Union (IoU) when ground truth meshes are available, the LI Chamfer Distance CD₁ (x (x10²) and euclidean distance based F-Score (FS) when ground truth points are available, and finally Normal Consistency (NC) when ground truth normals are available. We detail the expressions of these metries in the supplementary material.
4.3 Datasets
ShapeNet [12] consists various instances of 13 different synthetic 3D object classes. Following
[80], we use the first 20 shapes from the test split of each class for evaluation. We generate inputs of
different sizes (10k, 3k, 500) while adding gaussian noise of standard deviation 0.005 following the
literature (e.g. [7, S7D. ScanNet v2[18] is challenging real dataset containing 1513 room scans as captured with an RBG-D camera. We use 100 scans from the test split for evaluation. We sample 10k points from the scans as input. Faust [6] consists of real scans of 10 human body identities in 10 different poses. We use the entire 100 scans for evaluation. We sample 10k points from the scans as inputs. We use the provided mesh registrations to compute IoU.
4.4
Object Level Reconstruction
Table 1: IoU of ShapeNet reconstruction.
Table 2: CD₁ J of ShapeNet reconstruction.
Conv
SAC
NP
Poco
Ours
Conv
SAC
NP
Poco
Ours
Chair
0.87
0,88
0,67
0,90
0,91
Chair
0,41
0,39
0,56
0,42
0,34
Lamp 0,77
0,75
0,64
0,80
0,87
Lamp
0.47
0,44
0,72
0,56
0,42
Table
0,89
0,79
0,71
0,89
0,91
Table
0,35
0,56
0,62
0,38
0,30
Mean 0,84
0,81
0,67
0,86
0,90
Mean
0,41
0,46
0,63
0,45
0,35
Table 3: FS f of ShapeNet reconstruction.
Table NC f of ShapeNet reconstruction.
Conv
SAC
NP
Poco
Ours
Conv
SAC
NP
Poco
Ours
Chair
0,95
0,98
0,87
0,98
0,99
Chair
0,95
0,95
0,88
0,95
0,96
Lamp 0,92
0,94
0,83
0,95
0,96
Lamp
0,9
0,89
0,90
0,90
0,92
Table
0,98
0,92
0,80
0,98
0,99
Table
0,96
0,93
0,78
0,96
0,97
Mean
0,95
0,95
0,83
0,97
0,98
Mean 0,94
0,92
0,85
0,94
0,95
We show here reconstruction results from 10k sized noisy point clouds of classes Chair, Table and Lamp of ShapeNet. Convolutional occupancy models were trained with 3k sized noisy inputs. We report comparison on IoU in Table 1, CD; in Table 2, FS in Table 3, and NC in Table 4. Figures and 4 shows qualitative comparisons. Our method improves consistently across all metrics over its baseline Poco, as well as the other strategies. Methods that benefit from deep data priors (Cony, SAC, Poco, Ours) outperform data prior free overfitting (NP). We note also that these results show that the finetuning strategy SAC is not stable and does not improve consistently on its deep convolutional occupancy network Conv. This also confirmed in the visual results in Figure2, where the finetuning can lead to breakage in the reconstruction. Our method delivers more robust reconstruction generally with less holes and artifacts. As it can be seen in the same figure, it can particularly recover from failure cases of its bascline Poco. We believe that many of the failures of the bascline Poco are due to
7

---

Input
SAC Conv Figure 2: Qualitative comparison of ShapeNet reconstructions.
Ours
Poco
GT
its lack of generalization w.r.t. the input size. Figure 4 shows additional visual comparisons to SPSR. We provide numerical comparison to this classical method in the supplementary material.
4.5 Generalization to Real Scene Scans
GT
Poco SAP Figure 3: Qualitative comparison of ScanNet v2 reconstructions,
Ours
SAC Conv
Input
evaluate here the generalization from synthetic object training to real scenes. We show recon- struction results from 10k points sampled from real scans of ScanNet v2 in Table 5. As meshes are not available we can not compute loU here. Our method (Ours(Poco)) based on Poco outperforms the bascline and the competition, demonstrating superior generalization ability. Figure 3 shows few

---

Table 5: ScanNet reconstruction.
Table 6: Faust reconstruction.
CD
NC1
FSt
loUt
CD₁
NC1
FS
SPSR
2.27
0.74
0.68
SPSR
0.48
0.91
0.91
SAP
1.12
0,73
0,67
SAP
0,90
0,29
0,93
0,98
Conv
1,22
0,71
0,60
Conv
0,85
0,41
0,92
0,95
SAC
0,97
0,76
0,77
SAC
0,77
0,30
0,92
0,95
Poco
1,11
0.77
0,79
Poco
0,90
0,32
0,93
0,97
Ours (Poco)
0,94
0,79
0,80
Ours
0,92
0,26
0,95
0,99
Ours (Conv)
0,70
0,77
0,79
Figure 4: Qualitative comparison to Screened Poisson Surface Reconstruction (SPSR). Input Ours / SPSR, in this order.
qualitative results. While we offer improved performances compared to the competition, we note that the quality of our reconstruction remains limited still is such challenging setting. We also show that our method can be successfully applied to other existing occupancy networks, as combining our strategy with baseline Conv (Ours(Conv)) leads considerable improvement in its performance.
4.6 Generalization to Real Articulated Shape Scans
We evaluate here generalization from synthetic object training to real non-rigid human shapes. We show reconstruction results from 10k points sampled from real scans in Faust. As summa- rized Table 6, both convolutional networks SAP and Poco offer good initial performances. Notice how the finetuning based SAC can im- prove of its baseline (Conv) CD₁ conversely worsening its loU score. We outper- form both baseline and competing methods. Figure 5 shows visual comparison of the re- sults, where we recover from some failures of our baseline (Poco), and show limitations of the finetuning strategy in SAC. Figure 4 shows ad- ditional visual comparisons to SPSR.
while
4.7
Ablative Analysis
We conduct further ablative studies of pro- posed method on the Lamp class of ShapeNet, found its shapes be challenging and dis- playing many fine and complex features. Table 7 shows the performance of both our method and our baseline network Poco [7] for inputs of size 500, 3k and 10k. We note that Poco has been trained on 3k sized inputs. Notice first how the performance of Poco improves in CD, while deteriorating in IoU. This shows the performance of this model does not necessarily scale with input size. Our method on the other hand shows healthy consistent improvements over the baseline Poco, which also scale with input size. We notice also that the performance gap w.r.t. Poco is slightly tighter in the sparsest setup (500 points). We believe this due to the inductive bias of kernel methods favouring smoothness, as such prior can prove less effective for extremely sparse data.
Input
Ours
Poco
SAC
SAP
Figure 5: Qualitative comparison of Faust recon- structions.

---

Table 8 shows an ablation of the design choices of our method. Our performance drops as expected whether we disable the kernel hyperparameters 0 finetuning (w'o 0), the Nyström samples Z finetun- ing (w/ Z). We note also the importance of learning the regression in feature space. We notice that reconstruction often fails when we attempt to solve the regression in euclidean space. Hence our w/o Feat. version uses a Fourier positional encoding to stabilize this baseline. However, this version still under performs compared to our final model. Disabling the tuning all together (w/o tune) results in performance drop well. Increasing the number of number of Nyström samples in this context improves the performance.
Table 8 provides also an ablation the number of Nyström samples m while tuning (Ours). We find that generally the higher the number of inducing features the better the performance, as it was the case without hyper- parameterr optimization (w/o tune). This is expected as the approximation error of the Nyström KRR decreases with more inducing points. Additionally, more learnable inducing points implies more representation power for the KRR function. Increasing m comes however with increased computation overhead due to both solving the Nyström KRR with higher number of inducing features Z and also learning these features. Increasing m beyond certain limit can also lead to overfitting. As our aim this work is an efficient approach to improve generalization, we find a relatively small value (500) already sufficient to produce satisfactory shape predictions and offers a good performance/compute overhead trade-off.
Table 7: Ablation of input size.
Table 8: Ablation of our model.
IoUt
CD
NC
FSt
loUt
CD
NC1
FS
Poco(500)
0,71
0,85
0,86
0,84
w/o tune (m 1k)
0.84
0.61
0.92
0.96
Ours (500)
0,73
0,80
0,85
0,85
w/o tune (m 5kk
0.87
0.55
0.92
0.97
w/o tune (m = 10k)
0.87
0.53
0.92
0.96
Poco(3k)
0,85
0,60
0,91
0,96
Ours (3k)
0,86
0,52
0,92
0,96
w/o Feat.
0.45
2.23
0.79
0.58
w/o !
0.86
0.52
0.92
0.95
Poco[10k)
0,80
0,56
0,90
0,95
w/o
0.86
0.53
0.92
0.96
Ours (10k)
0,87
0,46
0,92
0,96
Ours (m = 500)
0.87
0.42
0.92
0.96
Ours (m 1000)
0.88
0.41
0.92
0.97
Ours (172 = 5000)
0.86
0.43
0.91
0.96
Finally, we note that we chose to use an equal number of input and augmented samples (Equation 5) empirically as it gave the best overall results. It is possible that for high levels of noise, higher number of augmented samples could prove beneficial. However, higher levels of noise for the input also imply less accurate predictions from the occupancy network, meaning less reliable pseudo-labels for the augmented samples.
5 Limitations
Extreme generalization from relatively sparse point cloud is still challenging for our method as well as the literature and we will seek further improvement in future work. Although it offers robustness, the inherent inductive bias in our method also favours smoothness, which can prove not ideal for representing the highest level of detail. Ablation of number of the NKRR Nyström samples in Table 8 shows that the KRR can possibly suffer from overfitting for large values of m, hence setting this hyperparameter can be challenging. We provide visualizations and short discussion of the behavior of our method under some baseline failures in the supplementary material.
6 Conclusion
We presented method for shape reconstruction from unoriented point cloud that combines the data prior of a preexisting deep reconstruction network and an efficient non-parametric interpolant. The resulting combination offers improved generalization finetuning, and several state-of-the-an methods. We hope we can inspire more work in the direction of fine-tuning and transfer learning from preexisting feedforward occupancy networks, especially since there are many competing strategies that could be explored. For instance, while we propose here to improve the boundary decision by learning the shape function, alternative strategy could be to tune solely the features instead.
the network baseline, network test-time
10

---

References
[1] Nina Amenta, Sunghee Choi, and Ravi Krishna Kolluri. The power crust, unions balls, and the medial
axis transform. CG. 2001. 3
[2]
Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In CVPR, 2020.
[3 Matan Atzmon and Yaron Lipman. Sald: Sign agnostic learning with derivatives. In ICML. 2020. 3
[4] Yizhak Ben-Shabat, Chamin Hewa Koneputugodage, and Stephen Gould. Digs: Divergence guided shape implicit neural representation for unoriented point clouds. In Proceedings of the JEEE/CVF Conference on
Computer Vision and Puttern Recognition,, pages 19323-19332, 2022. 3
[5] Fausto Bernardini, Joshua Mittleman, Holly Rushmeier, Claudio Silva, and Gabriel Taubin. The ball-
pivoting algorithm for surface reconstruction. TVCG. 1999, 3
[6] Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. FAUST: Dataset and evaluation for 3D mesh registration. In CVPR, 2014, 7
[7] Alexandre Boulch and Renaud Marlet. Poco: Point convolution for surface reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6302-630111300000 6314, 2022. 1, 3,
4, S. 6. 7.9
[8] Jonathan C Carr, Richard K Beatson, Jon B Cherrie, Tim J Mitchell, W Richard Fright, Bruce C. McCallum, and Tim R Evans. Reconstruction and representation of 3d objects with radial basis functions. In SIGGRAPH, 2001.
[9] Frédéric Cazals and Joachim Giesen. Effective Computational Geometry for Curves and Surfaces. 2006.
[10] Rohan Chabra, Jan Lenssen, Eddy lig, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In ECCV, 2020. 3 [11] Fric R Chan, Connor 7. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Lconidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123-16133, 2022. 3
[12] Angel X Chang. Thomas Funkhouser, Lconidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository.
arXiv preprint arxiv:1512.03012 2015. 1, 5, 6, 7
[13] Yinbo Chen and Xiaolong Wang. Transformers as meta-learners for implicit neural representations. In European Conference on Computer Vision, 2022. 3
[14] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
2020. 3
[15]
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling, In CVPR, 2019. 3
[16] Julian Chibane, Aymen Mir, and Gerard Pons-Moll. Neural unsigned distance fields for implicit function
learning. In NeurlPS, 2020. 3
[17] Julian Chibane and Gerard Pons-Moll. Implicit feature networks for texture completion from partial 3d data. In European Conference on Computer Vision, pages 717-725. Springer, 2020. 3, 5
[18] Angela Dai. Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017. 7
[19]
Boyang Deng. Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrry Hinton, and Andrea Tagliasacchi.
Cvxnet: Learnable convex decomposition. In CVPR, 2020. 3
[20] Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir Kim, Bryan Russell, and Mathieu Aubry. Learning elementary structures for 3d shape generation and matching. In NewrlPS, 2019. 3
[21] Philipp Erler. Paul Guerrero, Stefan Ohrhallinger, Niloy J Mitra, and Michael Wimmer. Points2surf
learning implicit surfaces from point clouds. In ECCV, 2020. 3,4
[22] Haoqiang Fan, Hao Su, and Loonidas Guibas. A point set generation network for 3d abject reconstruction
from a single image. In CVPR, 2017. 3
[23] Kyle Genova, Forrester Cole, Avncesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In CVPR, 2020.
[24] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser.
Learning shape templates with structured implicit functions. In ICCV, 2019..01... 3
[25] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization
for learning shapes. In JCML. 2020.
[26] Thibault Groueix, Matthew Fisher. Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-máché
approach to learning 3d surface generation. In CVPR, 2018. 3
[27] Gaël Guennehaud and Markus Gross. Algebraic point set surfaces. In ACM siggraph 2007 papers, pages
23-cs. 2007. 1,3
[28] John C Hart. Sphere tracing: A geometric method for the antialiased ray tracing of implicit surfaces. The Visual Computer, 1996. 3
[29] Jiahui Huang, Zan Gojeic, Matan Atzmon, Or Litany, Sanja Fidlor, and Francis Williams. Neural kernel surface reconstruction. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern
Recognition, pages 4369-4379, 2023. 4
[30] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text- guided object generation with dream fields. 2022. 3
11

---

[31]
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner. Thomas Funkhouser.
et al. Local implicit grid representations for 3d scenes. In CVPR. 2020.
[32]
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In CVPR, 2018. 3
[33]
Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. TOG, 2013. 1,3,
[34]
Ravikrishna Kolluri. Provably good moving least squares. TALG, 2008. 3
[35]
Tianyang Li, Xin Wen, Yu-Shen Liu. Hua Su, and Zhizhong Han. Learning deep implicit functions for 3d
shapes with dynamic code clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and
Pattern Recognition, pages 12840-12850, 2022. 3
[36]
Siyou Lin. Dong Xiao, Zuoqiang Shi, and Bin Wang. Surface reconstruction from point clouds without normals by parametrizing the gauss formula. ACM Transactions Graphics, 42(2):1-19, 2022.
[37] Stefan Lionar, Daniil Emtsev. Dusan Svilarkovic, and Songyou Peng. Dynamic plane convolutional occupancy networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer
Vision, pages 1829-1838, 2021. 3
[38]
Yaron Lipman. Phase transitions, distance functions, and implicit neural representations. In ICML, 2021. 3 Chen Liu, Jimei Yang, Duygu Ceylan. Ersin Yumer, and Yasutaka Furukawa. Planenet: Piece-wise planar reconstruction from a single rgb image. In CVPR, 2018. 3
[39]
[40]
Hsuch-Ti Derek Liu, Francis Williams, Alce Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. arXiv preprint arXiv:2202.08.0 08345, 2022. 3
[41] Minghua Liu, Xiaoshuai Zhang, and Hao Su. Meshing point clouds with predicted intrinsic-extrinsic ratio
guidance. In ECCV, 2020. 3
[42] Shi-Lin Liu, Hao-Xiang Guo, Hao Pan, Peng-Shuai Wang, Xin Tong, and Yang Liu. Deep implicit moving
least-squares functions for 3d reconstruction. In CVPR, 2021. 3
[43] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction
algorithm. In SIGGRAPH, 1987, 3,
[44] Baorui Ma, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces. In ICML,, 2021. 5,6 [45] Baorui Ma. Yu-Shen Liu, and Zhizhong Han. Reconstructing surfaces for sparse point clouds with on- surface priors. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6315-6325, 2022. 3
[46] Baorui Ma, Yu-Shen Liu, Matthias Zwicker, and Zhizhong Han. Surface reconstruction from point clouds by learning predictive context priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6326-6337. 2022. 3
[47] Giacomo Meanti, Luigi Carratino, Ernesto De Vito, and Lorenzo Rosasco. Efficient hyperparameter tuning for large scale kernel ridge regression. In International Conference on Artificial Intelligence and Statistics, pages 6554-6572. PMLR, 2022. 2, 6
[48] Giacomo Meanti. Luigi Carratino, Lorenzo Rosasco, and Alessandro Rudi. Kernll methods through the roof: handling billions of points efficiently. Advances Neural Information Processing Systems,
23:14410-14222 2020. 2, 5
[49] Corentin Mercier, Thibault Lescoat,tLesccatt Pierre Roussillon, Tamy Boubekeur, and Jean-Mare Thiery. Moving level-of-detail surfaces. ACM Transactions on Graphics (TOG), 1-(4) 2022. 3
[50] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEF/CVF conference on computer vivion and pattern recognition, pages 4460-4470, 2019. 3
[51] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.
Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 3
[52] Amine Ouasfi and Adnane Boukhayma. Few'zero zero level set' -shot learning of shape signed distance functions
in feature space. In ECCV, 2022. 3
[53] Amine Ouasfi and Adnane Boukhayma. Mixing-denoisee generalizable occupancy networks. In 3DV.
2024. 3
[54] David Palmer, Dmitriy Smirnov, Stephanic Wang, Albert Chern, and Justin Solomon. Deepeurrents: Learning implicit representations of shapes with boundaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18665-18675, 2022. 3
[55] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019. 3
[56] Songyou Peng. Chiyu Jiang, Yiyi Liao, Michael Niemeyer. Mare Pollefeys, and Andreas Geiger. Shape as points: A differentiable poisson solver. Advances in Neural Information Processing Systems, 34:13032- 13044, 2021. 3,
[57] Songyou Peng, Michacl Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision, pages 523-540. Springer, 2020. 3,
5. 6, 7
[58] Charles Qi, Hao Su. Kaichun Mo. and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017. 3
[59] Marie-Julie Rakotosaona, Noam Aigerman, Niloy Mitra, Maks Ovsjanikov, and Paul Guerrero. Differen-
tiable surface triangulation. In SIGGRAPH Asia, 2021. 3
[60] Germot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high
resolutions. In CVPR, 2017.
[61] Berharrd Schölkopf, Joachim Giesen, and Simon Spalinger. Kernel methods for implicit surface modeling. In NewrlPS, 2004. 3
12

---

[62] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-froms from-mmtion revisited. In Conference on
Computer Vision and Puttern Recognition (CVPR), 2016. 1
[63]
Johannes Lutz Schönberger. Enliang Zheng, Marc Pollefeys, and Jan-Michacl Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
[64] Vincent Sitzmann. Eric R Chan. Richard Tucker, Noah Snavely, and Gordon Wetzstein, Metasdf: Meta- learning signed distance functions. In NewrIPS, 2020. 3
[65]
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Neur!PS, 2020. 3
[66]
Vincent Sitzmann, Semon Rezchikov, William T Freeman, Joshua B Tenenbaum, and Fredo Durand, Light field networks: Neural scene representations with iinlleeealluaton rendering. In NewrIPS, 2021. 3 Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein. Scene representation networks: Continuous
[67]
3d- structure-aware neural scene representations. In NeurIPS, 2019. 3
[68]
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometrie level of detail: Real-time rendering with implicit 3d shapes. In CVPR, 2021. 3
[69] Jiapeng Tang. Jiabao Lei, Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang. Sa-convonet: Sign-agnostic optimization of convolutional occupancy networks. Proceedings of the IEEE/CVF International
Conference on Computer Vision. pages 6504-6513, 2021. 2, 6
[70] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient
convolutional architectures for high-resolution 3d outputs. In ICCV, 2017. 3
[71] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Christian Thcobalt. Patchnets: sitch-baeed generalizable deep implicit 3d shape representations, In ECCV, 2020. 3
[72] Shubham Tulsiani, Hao Su, Leonidas J. Guihas, Alexei A. Efros, and Jitendra Malik. Learning shape
abstractions by assembling volumetric primitives. In CVPR, 2017. 3
[73] Nanyang Wang. Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In ECCV, 2018. 3
[74] Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 106. 10689, 2021.
[75] Peng-Shuai Wang. Yang Liu, Yu-Xiao Guo, Chun-Yu Yu Sun, and Xin Tong. O-enn: Octree-hased convolu-
tional neural networks for 3d shape analysis. TOG, 2017. 3
[76] Shaofei Wang, Marko Mihajlovic, Qianli Ma, Andreas Geiger, and Siyu Tang. Metaavatar: Learning
animatable clothed human models from few depth images. Advances in Neural Information Processing
Systems, 34:28 10- (0-2822, 2021. 3
[77] Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines.
Advances in neural information processing systems, 13, 2000. 5
[78] Francis Williams, Zan Gojcic. Sameh Khamis, Denis Zorin. Joan Bruna, Sanja Fidler, and Or Litany. Neural fields as learnable kernels for 3d reconstruction. In CVPR, 2022. 3.4
[79] Francis Williams, Teseo Schneider, Claudio Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. Deep geometric prior for surface reconstruction. In CVPR, 2019. 3
[80] Francis Williams, Matthew Trager, Joan Bruna, and Denis Zorin. Neural splines: Fitting 3d surfaces with infinitely-wide neural networks. In CVPR. 2021. 3.
[81] Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum. learning a probabilistic latent space of object shapes via 3d generative-advvesaree
modeling. In NeurlPS, 2016. [82] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.
3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
[83] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-488
2021. 3
[84] Mohsen Yavartanoo, Jacyoung Chung, Reyhanch Neshatavar, and Kyoung Mu Lec. 3dias: 3d shape
reconstruction with implicit algebraic surfaces. In /CCV, 2021. 3
[85] Junsheng Zhou, Baorui Ma, Liu Yu-Shen, Fang Yi, and Han Zhizhong. Learning consistency-aware unsigned distance functions progressively from raw point clouds. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3
[86] Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, and Derek Hoiem. 3d-prnn: Generating shape primitives with recurrent neural networks. In CVPR, 2017.
13


Reviews:
[{"review_id": "iWe7Q3O9iy", "reviewer": "Reviewer_LSLk", "summary": "This paper aims to reconstruct smooth and high-quality surface from noisy point cloud. While previous works like Poisson surface reconstruction can extract surface from point cloud, but it's sensitive to input noise. The existing learning-based methods are able to handle noisy point cloud, but often suffer from generalizing to other dataset, real-world scenes, and large scale point cloud. The authors propose to leverage pretrained learning-based occupancy network (POCO) for extracting point cloud features and generate pseudo label for new points. The Kernel Ridge Regression (KRR) and the Nystrom samples are jointly optimized with gradient descent, representing the final shapes from the input point cloud. The experiments show that proposed method achieve better evaluation metrics (CD, IoU, NC, FS) on synthetic ShapeNet, real-world dataset ScanNet v2. The qualitative results and ablation study are conducted to verify the design choices. ", "questions": "- What's the fundamental difference between the proposed method and previous kernel-based neural surface reconstruction (see the references in the limitation section)?\n- How could the optimization recover the shape surface from point cloud if the POCO backbone is not able to produce reasonable prediction?\n- Is there any intuition behind the design choice where the authors decide to represent the surface as occupancy instead of signed distance function?\n\n### Other comments\n- Missing section reference in line 42\n- typo in line 148 \n- Missing bracket in eq. 12", "limitations": "There's no potential negative societal impact of this paper. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The quantitative results in Table 1-6 demonstrate that proposed method achieve better evaluation metric, including Chamfer distance, IoU, normal consistency, and F-sore. These metrics indicate the reconstructed meshes have superior quality.\n- The qualitative results in the figures show that the reconstructed surface is smoother and more complete, in contrast to the baseline methods.\n- The proposed method can generalize to synthetic and real-world large scale input point cloud.", "weaknesses": "- Formulating implicit surface with kernel-based method is not a new idea. The fundamental difference between proposed method and previous works \"Neural Kernel surface reconstruction\", \"Neural Fields as Learnable Kernels for 3D Reconstruction\" is not described in the paper, and the comparison is also missing in the experiments section, making it difficult to evaluate the technical contribution.\n- The propose method relies on pretrained POCO to generate point features and pseudo points label. The authors argue that the learning-based methods suffer from generalizing to out-of-distribution data. As a result, the proposed method is likely to fail for the data that POCO cannot produce good enough initialization (e.g. incorrect occupancy prediction). As a result, the capability of generalization of proposed method does not outperform previous learning-based method significantly. \n- Per-scene optimization is required to get KRR hyper-parameters and Nystrom sample points. While the optimization doesn't take much time, the comparison with baseline methods might be unfair since most of them only need single forward pass. In addition, according to the quantitative results in the tables, the number only improve slightly from POCO baseline.\n- The weakness of previous works and difference of proposed method is not clear in the related works section. To be more specific, in the subsection \"Unsupervised Implicit Neural Reconstruction\", various previous works are mentioned along with some technical details, but the description about the their common limitation is missing, the only text that describe this group of works is \"All of the aforementioned work benefits from efficient gradient computation through back-propagation in the neural network.\" which does not contain much information. \n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### “What's the fundamental difference between the proposed method and previous kernel-based neural surface reconstruction?”:\n\nCVPR 23 Proceedings were not available yet at the Neurips 23 deadline. Existing learning based kernel reconstruction methods ([78] Neural splines CVPR 2021, [76] Neural kernel fields CVPR 2022) differ from our work in the following:\n- Both [76] and [78] address the problem of reconstruction from points and their normals. Differently, we use KRR to reconstruct from points only which is a much harder problem.\n- Differently from [76] and [78],  we propose a strategy to reduce the generalization gap by  solving our regularized ERM problem (Equation 6)  in a RHKS  space  adapted to  the shape we would like to recover. This is possible thanks to the unique correspondence between RKHS and kernels. Hence, instead of relying on handmade fixed kernels as in  [76] and [78], we learn the kernel parameters using   loss function that avoids overfitting on the data  term (Equation 10). Additionally, we can  dynamically adjust this function within the reproducing kernel Hilbert space by learning the Nystrom  samples. This is a novel approach that was not explored in the context of shape reconstruction , as the  resulting function from this optimization is more complex and expressive then a simple KRR mapping   [76],78]. We will clarify further relation to [76] and [78] in our related work section.\n\n### “How could the optimization recover the shape surface from point cloud if the POCO backbone is not able to produce reasonable prediction?”:\n\nFeedforward generalizable occupancy networks such as Poco can underfit the input point cloud as the knowledge that input samples are on the surface is not explicitly enforced. One solution to improve the prediction of these networks is to ensure the input points coincide with the level set of the predicted shape function. One existing way to do this is finetuning the network as in SAC [7] with such a loss. We find this strategy to be unstable. It can even lead to worst results than the baseline in many cases (see Results section). One of the reasons for this is the difficulty of optimizing a large number of neural network parameters. Alternatively (Line #173), we propose to optimize less parameters under more constraints which yields more robustness, as our learnable KRR offers both flexibility and spatial regularization to the shape function. In the rebuttal PDF, Figure 2, we show examples where the network baseline fails while we can recover good reconstructions. \n\n### “Is there any intuition behind the design choice where the authors decide to represent the surface as occupancy instead of signed distance function?”:\n\nThis choice was based on the fact that the most common generalizable feedforward implicit reconstruction from point cloud networks (e.g. Conv[55], Poco[7]) use the occupancy representation.\n\n### “Per-scene optimization is required to get KRR hyper-parameters and Nystrom sample points. While the optimization doesn't take much time, the comparison with baseline methods might be unfair since most of them only need single forward pass”:\n\nWe compare to both feedforward baselines, and test-time optimization methods such as ours, and we outperform them all. Optimization competition includes the closest existing method to ours, based on finetuning a generalizable occupancy network at test time SAC [7], and a data prior free sota method NP [43].\n\n### “In addition, according to the quantitative results in the tables, the number only improve slightly from POCO baseline”:\n\nWe beg to differ. We find our improvements over our baseline to be important and consistent across all 3 datasets and metrics. For the most important metric i.e. Chamfer distance (CD1), we offer an improvement of 22% in ShapeNet (Table 2), 15% in ScanNet (Table 5) and 19% in Faust (Table 6).\n\n### “The weakness of previous works and difference of proposed method is not clear in the related works section”:\nWe will improve the related work discussion in the next version."}, {"review_id": "bxG8NpNq1a", "reviewer": "Reviewer_JKo3", "summary": "This paper proposes an efficient mechanism to address the generalization issues often encountered by generalizable feedforward models trained for implicit shape reconstruction from unoriented point clouds. The solution proposed in this work is to use kernel ridge regression in the network's feature space, tuned by gradient descent. At a high-level, this approach utilizes a pre-trained shape reconstruction network to produce pseudo-labels to provide more supervision. The effectiveness of the proposed method is demonstrated on multiple datasets, including synthetic and real data.", "questions": "1. What does the k_theta in L184 represent?", "limitations": "The paper has discussed its limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The overall quality of this paper is high. The sections are well-written and cohesively presented, ensuring a smooth flow of information.\n2. The extensive experiments effectively highlight the underfitting and overfitting issues that this research seeks to address. The motivation for the study is convincingly articulated. \n3. The decision to implement kernel regression in the feature space of a pre-trained occupancy network is sound and logical. Even though the pre-trained occupancy network is already robust and strong, the approach taken in this paper is understandable given its objective to create a test-time augmentation method.\n", "weaknesses": "1. Comparative Analysis: Given that the proposed method is primarily a test-time optimization technique, it would be beneficial to include a comparison with more test-time augmentation methods. While the authors do mention a test-time fine-tuning baseline in lines 206-207, it might be worth investigating if there are other advanced test-time optimization techniques, specifically ones that could directly apply to Poco features.\n\n2. Demonstrating Generalizability: The results showcased in this paper are primarily based on the Poco network, which already significantly outperforms other baseline methods as demonstrated in Table 1. Although I believe this method could be applied to other feature extractors, it would strengthen the paper to present more numerical results to support this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### “What does the k_theta in L184 represent?”\n\nAs reported line #217,  k_theta is the Gaussian kernel with a single length-scale for each data-dimension in our experiments. We will clarify this. We note that by using a Gaussian kernel we benefit from the universality properties of the associated (RKHS) i.e. a hypothesis space rich enough to approximate any continuous function arbitrarily well. \n\n### \"W1. Comparative Analysis: Given that the proposed method is primarily a test-time optimization technique, it would be beneficial to include a comparison with more test-time augmentation methods. While the authors do mention a test-time fine-tuning baseline in lines 206-207, it might be worth investigating if there are other advanced test-time optimization techniques, specifically ones that could directly apply to Poco features.\"\n\nApart from the work in SAC[7] and ours, we are not aware of the existence of other work in the area of fine-tuning or transfer learning from preexisting feedforward occupancy networks. We hope we can inspire more work in this direction, especially since there are many competing strategies that could be explored. For instance, while we propose here to improve the boundary decision by learning the shape function, an alternative strategy could be to tune solely the features. It's important to note that such a feature-based strategy might prove computationally expansive for large point cloud networks.\n\n### \"W2. Demonstrating Generalizability: The results showcased in this paper are primarily based on the Poco network, which already significantly outperforms other baseline methods as demonstrated in Table 1. Although I believe this method could be applied to other feature extractors, it would strengthen the paper to present more numerical results to support this.\"\n\nOur method can be successfully applied to other existing generalizable occupancy networks. To support this, we show the result of combining the Shapenet trained network in Conv [55] with our strategy in Table 2 in the rebuttal PDF. We perform this experiment in the ScanNet benchmark as it is the most challenging in terms of generalization. Our Conv [55] based method outperforms its baseline. We will add this result in our next version."}, {"review_id": "X2uXIfy9Ta", "reviewer": "Reviewer_6hDU", "summary": "This paper aims at the task of implicit surface reconstruction from noisy point clouds. It tackles the model generalizability and robustness by introducing a kernel ridge regression as a implicit function that maps a shape feature to occupancy. \n", "questions": "How the ground truth point clouds of the real dataset (ScanNet v2 and Faust) acquired? What is the difference between ground truth point clouds and noisy point clouds for these real datasets. Are the noise introduced during scanning or added afterwards additionally?\n", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "\nThe proposed method achieve higher accuracy compared to prior surface reconstruction method. It shows certain robustness when applied in real datasets. \n", "weaknesses": "\na. What is the motivation behind KRR? How does it help the generalizability of the models? Why it is fitted for surface reconstruction from noisy point clouds? The authors should demonstrate the intuition behind the introduction of KRR, compared to other occupancy decoders.\nb. From the results on ShapeNet, we see that even for test dataset that have the same density and synthetic/real property as the training dataset, the proposed method still outperforms other methods with a notable margin. Why is that?\nc. The writing is not very clear, especially how the introduced KRR is integrated in a deep learning network. Is the feature encoder fixed in the proposed method? Is the optimization of KRR conducted for each test sample? If so, the experiments need to compare the test time and accuray simultaneously. Also I have confusion on the discussion from L136-L147. How the intrinsity or extrinsity of point feature could effect the proposed method?  Besides, Fig. 1 is not organized well. There too many notations and unclear text color, which does not help understanding. \n\nTypos:\nL42: Wrong reference\nL45: ; -> ,\nL136: isntance -> instance\nL148: and MLP -> an MLP\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### \"a.What is the motivation behind KRR? How does it help the generalizability of the models? Why it is fitted for surface reconstruction from noisy point clouds? The authors should demonstrate the intuition behind the introduction of KRR, compared to other occupancy decoders.\"\n\nWhile feedforward generalizable occupancy networks can offer very good performances and fast inference, they can underfit the input point cloud as the knowledge that those samples are on the surface is not explicitly enforced. One solution to improve this aspect (line #39) and hence the generalization of these networks at test time, is to fine tune the network as in SAC [7] with such a loss. We find this strategy to be unstable and can even lead to worst results than the baseline in many cases (see Results section). One of the reasons for this is the difficulties arising from optimizing a large number of neural network parameters.\n\nThese instabilities  highlight the necessity for regularization in this adaptation process. We need to balance  the need to maintain the expressive capacity of the learned functions while ensuring stability and robustness. To this end we devise three important design choices. \nFirstly, we restrain the hypothesis space of the shape functions to be a reproducing kernel Hilbert space. (RKHS).  By the Representer Theorem, the minimizer of our regularized empirical risk minimization (ERM) problem (equation 6 in the paper)  emerges naturally as the solution of a KRR problem. \nSecondly, by using a Gaussian kernel we benefit from the universality properties of the associated (RKHS) ie. a hypothesis space rich enough to approximate any continuous function arbitrarily well.  \nThirdly,  we propose a strategy to solve our regularized ERM problem (Equation 6)  in a RHKS  space  adapted to  the shape we would like to recover. This is possible thanks to the unique  correspondence between RKHS and kernels. Hence, instead of relying on handmade fixed kernels, we learn the kernel parameters using a loss function that avoids overfitting on the data term (Equation 10).\n\nThus, the primary motivation behind KRR  is to  avoid  the difficulties arising from optimizing a large number of neural network parameters, as in SAC, while maintaining the expressive capacity necessary for effective shape modeling. \n\n\n### “b. From the results on ShapeNet, we see that even for test dataset that have the same density and synthetic/real property as the training dataset, the proposed method still outperforms other methods with a notable margin. Why is that?”:\n\nAs it has been noted by previous work too [76,7], feedforward generalizable occupancy networks can underfit the input point cloud, i.e. there is no explicit guaranty that input points are at the predicted shape function level set from a single forward pass. Enforcing this constraint under well chosen regularizations (i.e. kernel prior) leads to improvements at test time. \n\n### “c. The writing is not very clear, especially how the introduced KRR is integrated in a deep learning network”:\n\nWe will clarify these details and improve the writing in the next version.\n\n### “c. Is the feature encoder fixed in the proposed method?”:\nYes.\n\n### “c. Is the optimization of KRR conducted for each test sample? If so, the experiments need to compare the test time and accuracy simultaneously.”\n\nYes, as it the case for test time optimization competition presented in the paper (e.g. SAC, NP). As reported line #213, for a 10k sized input point cloud, our method takes around 10 secs to converge on a NVIDIA RTX A6000 GPU. Other presented test time optimization methods SAC [67]: 60 secs, NP [43]: 300 secs, using their available pytorch implementations.\n\n### “c. How the intrinsity or extrinsity of point feature could effect the proposed method?”:\n\nWe build here on a method with an intrinsic encoder (Poco [7]) as it offers good performance across many benchmarks. We show in Table 2 in the rebuttal PDF that our approach can be applied successfully to a method with an extrinsic encoder as well (Conv [55]).\n\n### “c. Fig. 1 is not organized well.”\n\nWe will improve the figure, and explain the color coding more thoroughly."}, {"review_id": "d872Z04Bvf", "reviewer": "Reviewer_7xQW", "summary": "The paper proposes a method for improving shape reconstruction from a set of sparse and noisy unoriented point clouds. As data-driven trained networks can have failures on out-of-distribution examples, this method proposes to take the input point cloud, the predicted surface points, and the features generated by the pre-trained network, and estimate a non-parametric set of kernels to represent the output shape occupancy. Results on shape net, Faust, and scannet show improved reconstruction performance compared to prior deep-network only approaches, and recent work that perform test-time fine-tuning. ", "questions": "[Q1] Why is robustifying pre-trained shape reconstruction networks the desired approach to improving shape reconstruction, instead of directly designing a reconstruction method with kernels [1]?  What benefit does this alternative direction offers - why post-correct rather than inducing these priors directly in the representation and in the original training. What is the main novelty and contributions over neural kernel fields, and neural kernel surface reconstruction?  \n[1] Neural Kernel Surface Reconstruction CVPR 2023 (and its predecessor neural kernel fields CVPR 2022). The related work cites NKF in “Reconstruction from Point Cloud,” but does not describe how the submitted paper fits in this context. \nThe introduction lacks motivation other than describing the limitations of existing deep-learning based shape reconstruction methods. \n", "limitations": "See  W3, while they are mentioned, I feel limitations was not adequately addressed with examples. For a visual topic like shape reconstruction, examples would help better illustrate them. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- [S1] The method outperforms prior work it compares in SoTA deep network shape prediction and reconstructs well thin surfaces. It was nice to see strong generalization results on human shape reconstruction trained only from shapenet data. \n- [S2] The idea of inducing additional priors and structure in shape prediction is a nice direction to generalization", "weaknesses": "- [W1] The related work covers shape representations, optimization-based reconstruction, and feed-forward prediction (either supervised or unsupervised). No context is provided of how this work fits into the field. See Q1 below, the paper needs more motivation and why this particular approach of post-processing using a pre-trained network is a fruitful direction to pursue for improving reconstruction from sparse and noisy point clouds.  \n- [W2] The paper is missing a baseline to classical methods such as PCA-neighbor-based normal estimation + screened-poisson surface reconstruction, using input points and optional pseudo-points from the network prediction. \n- [W3]  No qualitative visuals of failures provided, or discussion of other limitations, such as the assumption that the input point cloud is a contiguous surface and that kernel prior of locally smooth surfaces doesn’t always hold (not just recovering details) - how does the reconstruction perform on an input of several nearby rocks with jagged surfaces and holes?  I wish the paper also showed more qualitative examples - only one from scannet and three from Faust are shown.  \n\nMinor weakness: \n- Multiple typos, for example section 3.1: (L136 isntance —> instance, L138 contenting —> contending, L148 and —> an, L153 order-invariantly —> pooled to be order invariant, L154 “Without loss of generality we build in this work on the model “). \n- Writing style - some parts of the text felt extraneous. If the method focuses on post-processing, why devote L136-L147 talking about different input network architecture designs. It seems the proposed method can work with most pretrained-network shape reconstruction methods, as long as it can provide pseudo-labels and features about the 3D space points. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### “Q1/W1. Why is robustifying pre-trained shape reconstruction networks the desired approach to improving shape reconstruction, instead of directly designing a reconstruction method with kernels ? What benefit does this alternative direction offers - why post-correct rather than inducing these priors directly in the representation and in the original training”:\n\n- While building generalizable occupancy networks including kernel priors is definitely a promising direction, one might not have the important resources, the time and data required to build such models. Alternatively, given readily available generalizable shape models, we propose here an efficient solution to improve their generalization at test time instead. We believe both directions to be practical and useful for the community.\n\n- Rather than using a simple KRR, depending on manually crafted fixed kernels, we explore in this work the hyper-parameter space of Nystrom-KRR in the context of shape reconstruction by learning the inducing points  and the kernel parameters. Consequently we adapt our hypothesis space in accordance to the shape we aim to recover. It is not straightforward to integrate this process within the training of a generalizable network, and this is a challenge we are looking forward to uncover as part of future work.\n\n### “What is the main novelty and contributions over neural kernel fields, and neural kernel surface reconstruction?”:\n\nCVPR 23 Proceedings were not available yet at the NeurIPS 23 deadline. Existing learning based kernel reconstruction methods ([78] Neural splines CVPR 2021, [76] Neural kernel fields CVPR 2022) differ from our work in the following:\n- Both [76] and [78] address the problem of reconstruction from points and their normals. Differently, we use KRR to reconstruct from points only which is a much harder problem.\n- Differently from [76] and [78],  we propose a strategy to reduce the generalization gap by  solving our regularized ERM problem (Equation 6)  in a RHKS  space  adapted to  the shape we would like to recover. This is possible thanks to the unique correspondence between RKHS and kernels. Hence, instead of relying on handmade fixed kernels as in  [76] and [78], we learn the kernel parameters using a loss function that avoids overfitting on the data  term (Equation 10). Additionally, we can  dynamically adjust this function within the reproducing kernel Hilbert space by learning the Nystrom  samples. This is a novel approch that was not explored in the context of shape reconstruction , as the  resulting function from this optimization is more complex and expressive then a simple KRR mapping   [76],78]. We will clarify further relation to [76] and [78] in our related work section.\n\n\n### “[W2] The paper is missing a baseline to classical methods such as PCA-neighbor-based normal estimation + screened-poisson surface reconstruction”:\nFollowing the request of the reviewer, we provide an additional classical baseline (PCA normals + SPSR) in the rebuttal PDF (Figure 1, Table 1). We provide quantitative and qualitative comparison. This baseline deteriorates under low input point density and due to the inaccuracies of nearest neighbor based normal approximation.  \n\n### “[W3] No qualitative visuals of failures provided, or discussion of other limitations, such as the assumption that the input point cloud is a contiguous surface and that kernel prior of locally smooth surfaces doesn’t always hold (not just recovering details) - how does the reconstruction perform on an input of several nearby rocks with jagged surfaces and holes?”:\nWe addressed in this work reconstruction from uniform point cloud as it is the case in our competition work. We defer more challenging inputs such as partial, incomplete and extremely noisy point clouds to future work. We note that while we restrain our hypothesis space to a RHKS,  we ensure this space to be rich enough to approximate any continuous function arbitrarily well by using a gaussian kernel. Learning the hyperparemters of the Nystrom KRR including the kernel parameters and the Nystrom samples  allows as to solve the KRR in a RKHS specialized to the  shape we aim to recover. In addition,  we can dynamically adjust inputs (Nystrom samples) in case they are noisy or not placed conveniently initially wrt the underlying surface. As mentioned in  line #287 as part of limitations, kernels do favor smoothness which can prove tricky for recovering highly detailed shapes. We focus in this work on the robustness of the reconstruction rather than finer levels of detail. \nWe provide additional limitation in the rebuttal PDF and we will discuss them further in the next version. First ablation of number of nystrom samples m in Table3 shows that the KRR can possibly suffer from overfitting for large values of m, hence the choice of this hyperparameter can be challenging. Additionally, Figure 2 show an example where, despite improving on the reconstruction of the network baseline, it is still challenging for our method to recover the finest structures and intricate details of a complex shape. \n\n### “I wish the paper also showed more qualitative examples - only one from scannet and three from Faust are shown”:\nWe provide additional reconstructions in the supplementary material and the Rebuttal PDF."}, {"review_id": "EUMnKcLMyg", "reviewer": "Reviewer_Pm8Q", "summary": "To address the poor performance of feed-forward models on noisy real data, the authors propose using test-time Kernel Ridge Regression (KRR) to fine-tune the results for implicit shape networks, specifically in the context of recovering 3D reconstruction surfaces from noisy point clouds. The method involves incorporating a feedforward network called Poco and utilizing the input point cloud and the network's approximated logits to determine the optimal KRR for representing the final implicit shape. This method improves the performance of Poco.", "questions": "See weakness", "limitations": "The ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces an additional post-processing step, utilizing information from the input test data to enhance surface reconstruction and improve generalization ability for unseen cases. The idea is novel, as it combines a non-parametric model with a neural network to enhance the performance of the latter further.\n\n2. The authors have evaluated their method on various datasets to assess the entire pipeline.", "weaknesses": "1. The paper does not mention the overhead of the current method. Since the KRR model needs to be optimized for different point cloud inputs, it is crucial to report the optimization time in this paper.\n\n2. Currently, the training samples for KRR consist of an equal number of noisy input and network-predicted samples. Could the authors explain the reasoning behind this decision? It would be helpful to understand why they did not include more or fewer network-predicted samples. In cases where the input data is highly noisy, the inclusion of more network-predicted samples might prove more beneficial.\n\n3. The number of Nystrom Samples, denoted as \"m,\" should have an impact on the final output performance. However, the paper lacks a clear discussion on this aspect. In Table 8, there is an option labeled \"without tune\" with different values of \"m.\" Does this mean that there was no optimization performed on theta and Z? It would be better to include a comparison with optimization and different values of \"m\" for a comprehensive analysis.\n\n4. Some typos: broken reference in line 41; Eq. 5, for the input point cloud, the logit should be 0.5, on the surface?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### “1. The paper does not mention the overhead of the current method. Since the KRR model needs to be optimized for different point cloud inputs, it is crucial to report the optimization time in this paper”:\n\nAs reported line #213, for a 10k sized input point cloud, our method takes around 10 secs to converge on a NVIDIA RTX A6000 GPU. Other presented test time optimization methods SAC [67]: 60 secs, NP [43]: 300 secs, using their available pytorch implementations.\n\n### “2. Currently, the training samples for KRR consist of an equal number of noisy input and network-predicted samples. Could the authors explain the reasoning behind this decision? It would be helpful to understand why they did not include more or fewer network-predicted samples. In cases where the input data is highly noisy, the inclusion of more network-predicted samples might prove more beneficial”:\n\nWe chose this ratio empirically as it gave us the best overall results. We note that we experimented in this paper within the standard reconstruction benchmarks . It is possible indeed that for even higher levels of noise, a higher number of augmented samples could prove beneficial. However, higher levels of noise for the input also imply less accurate predictions from the occupancy network, meaning less reliable pseudo-labels for the augmented samples. We will include a discussion in the next version.\n\n### “3. The number of Nystrom Samples, denoted as \"m,\" should have an impact on the final output performance. However, the paper lacks a clear discussion on this aspect. In Table 8, there is an option labeled \"without tune\" with different values of \"m.\" Does this mean that there was no optimization performed on theta and Z? It would be better to include a comparison with optimization and different values of \"m\" for a comprehensive analysis.”\n\nFollowing the request of the reviewer we provide an ablation of parameter “m” with optimization in the rebuttal PDF in Table 3. We find that the higher the number of inducing features Z the better the performance, as it was the case without hyper-parameter optimization (Table 8 w/o tune). This is an expected result as the approximation error of the Nyström KRR decreases with more inducing points. On top of that, more learnable inducing points implies more representation power for the KRR function. Increasing m comes however with increased computation overhead due to both solving the Nyström KRR with a higher number of inducing features Z and also learning these features. Increasing the number of learnable inducing features beyond a certain limit can also lead to over-fitting. As our aim in this work is an efficient approach to improving generalization, we find a relatively small m value (500, line #221) already sufficient to produce satisfactory shape prediction results and offers a good performance/compute overhead trade-off. We will include the ablation table and a thorough discussion of this point in the next version."}, {"review_id": "sGFIC7OJGj", "reviewer": "Reviewer_q2Vh", "summary": "The paper presents a novel shape reconstruction method (point cloud-> reconstructed implicit shape) that combines a deep shape prior and a non-parametric interpolant. Existing deep shape reconstruction methods are typically trained on synthetic datasets (to obtain full supervision) and suffer from a domain shift when applied to real examples. In addition, they are sensitive to the number of points in the input point cloud. To address scaling, the paper proposes to use Kernel Ridge Regression (KRR), a non-linear interpolant that adds the condition that the points input at test time need to lie on the output implicit surface. ", "questions": "The method seems to rely on a pretrained occupancy network. Does this affect the performance or expressiveness of the technique? \n\nWhat is the computational time of the proposed method? From Algorithm 1, it appears to be iterative. \n\nHow sensitive is the method to the percentage of points that covers in the input, i.e., what if only a very partial view is provided? \n", "limitations": "The key contribution of the paper is that improves the synthetic to real gap, however, the few results on real data (sections 4.5 and 4.6) seems to indicate that performance is still significantly limited. It would help to identify specific example cases where the proposes method offers a significant advantage.  ", "rating": 5, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "A novel method for implicit shape reconstruction that combines deep shape priors and implicit shape outputs is proposes. The technique is innovative and generates successful reconstructions on public benchmarks (ShapeNet, FAUST, and ScanNet) and outperforms three public baselines (Poco, Conv, SAP, SAC). Detailed ablation studies on the number of input points and KRR design choices (tuning, number of samples). ", "weaknesses": "The key weakness of the works is that the paper could be written more clearly. The methodology section has several grammatical errors and is hard to follow. It would be helpful to have more clarity and explanations for a novice reader. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "###  “The paper could be written more clearly”:\n\nWe will improve the writing of the paper for the next version.\n\n### “The method seems to rely on a pretrained occupancy network. Does this affect the performance or expressiveness of the technique”:\n\nThe very objective (line #5 #39) of this work is to efficiently improve the generalization of a readily available occupancy network (typically priorly trained on large data using substantial resources). While our final performance is affected by our network baseline, we show in our experiments that we consistently outperform this baseline across all metrics and all 3 data-sets. Figure 2 in the rebuttal PDF shows an example where our baseline Poco fails while we can recover, thanks to the re-injection of the condition that inputs are at the levelset at test time. Furthermore, as requested by reviewer JKo3, we show in Table 2 in the PDF that our approach can be successfully applied to other existing occupancy networks (Conv [55]) as well besides Poco [7].\n\n### “What is the computational time of the proposed method? From Algorithm 1, it appears to be iterative”:\n\nAs reported line #213, for a 10k sized input point cloud, our method takes around 10 secs to converge on a NVIDIA RTX A6000 GPU. Other presented test time optimization competition SAC [67]: 60 secs, NP [43]: 300 secs, using their available pytorch implementations.\n\n### “How sensitive is the method to the percentage of points that covers in the input, i.e., what if only a very partial view is provided?”:\n\nWe did not address here the problem of reconstruction from incomplete or partial point cloud and we plan to do this in future work. We note that addressing this problem using our strategy would require using special generalizable occupancy networks that were trained specifically with such simulated partial input scenario. \n\n### “The key contribution of the paper is that improves the synthetic to real gap, however, the few results on real data (sections 4.5 and 4.6) seems to indicate that performance is still significantly limited”: \n\nWe improve both the gap caused by change in input density (Tables 1,2,3,4,7) and the synthetic to real gap (Tables 5,6). We outperform common state-of-the-art and improve considerably and consistently on our baseline on 3 datasets. For the most important metric i.e. Chamfer distance (CD1), we offer an improvement of 22%in ShapeNet (Table 2), 15% in ScanNet (Table 5) and 19% in Faust (Table 6) wrt to our occupancy network baseline. Figure 2 in the PDF shows additional examples where our method can recover from failures of our baseline.  We do not claim that we solve the generalization problem as it is a notoriously hard one, but we propose a novel and efficient way to approach it in the context of implicit shape reconstruction from point cloud, that also achieves very encouraging results. "}]

Author rebuttal:
We thank the reviewers for their valuable feedback. We will improve writing in the next version and correct typos pointed out by reviews. We will add the additional discussions and results provided in the rebuttal. We address the questions separately in the respective rebuttals and we summarize also bellow points that were important or common between several rebuttals. We provide the corresponding figures and tables in the PDF file.
 
### Motivation and Comparison to existing kernel reconstruction methods [76,78]

Our objective is to reconstruct the shape  by dynamically adjusting the network to the input point-cloud during testing. The instability observed  in SAC [7]  highlights the necessity for regularization in this adaptation process. We need to balance  the need to maintain the expressive capacity of the learned functions while ensuring stability and robustness. To this end we devise three important design choices. 

Firstly, we restrain the hypothesis space of the shape functions to be a reproducing kernel Hilbert space. (RKHS).  By the Representer Theorem, the minimizer of our regularized empirical risk minimization (ERM) problem (equation 6 in the paper)  emerges naturally as the solution of a KRR problem. 

Secondly, by using a Gaussian kernel we benefit from the universality properties of the associated (RKHS) ie. a hypothesis space rich enough to approximate any continuous function arbitrarily well. Thus, we avoid  the difficulties arising from optimizing a large number of neural network parameters, as in SAC, while maintaining the expressive capacity necessary for effective shape modeling. 

Thirdly, we propose a strategy to solve our regularized ERM problem in a RHKS  space  adapted to  the shape we would like to recover. This is possible thanks to the unique  correspondence between RKHS and kernels. Hence, instead of relying on handmade fixed kernels as in [76,78] , we learn the kernel parameters using a loss function that avoids overfitting on the data term (Equation 10 in the paper).

Regarding existing kernel reconstruction methods  we further note that: 

- Both [76] and [78] address the problem of reconstruction from points and their normals. Differently, we use KRR to reconstruct from points only which is a much harder problem.
- Differently from [76] and [78], we propose a strategy to improve  generalization  by solving our regularized ERM problem (Equation 6) in a RHKS space adapted to the shape we would like to uncover This is possible thanks to the unique correspondence between RKHS and kernels. Hence, instead of relying on handmade fixed kernels as in [76] and [78], we learn the kernel parameters using a loss function that avoids overfitting on the data term (Equation 10). Additionally, we can dynamically adjust this function within the reproducing kernel Hilbert space by learning the Nystrom samples. This is a novel approach that was not explored in the context of shape reconstruction, as the resulting function from this optimization is more complex and expressive then a simple KRR mapping [76],78]. We will clarify further relation to [76] and [78] in our related work section.

### Why improve on an existing occupancy network ? 

One might not have the important resources, time and data required to train a generalizable model from scratch. Alternatively, given readily available pretrained networks, we propose here an efficient and robust solution to improve their generalization at test time. We believe this strategy to be practical and useful for the community, and we hope this work will spur more contributions in the area of fine-tuning and transfer learning for shape reconstruction.  

### Are improvements marginal ? 

We outperform common state-of-the-art and improve considerably and consistently on our baseline on 3 datasets across all metrics. For the most important metric i.e. Chamfer distance (CD1), we offer an improvement of 22%in ShapeNet (Table 2), 15% in ScanNet (Table 5) and 19% in Faust (Table 6) wrt to our occupancy network baseline. 

### Can we use other generalizable occupancy networks ? 

Our method can be successfully applied to other existing occupancy networks. We show the result of combining the Shapenet trained network in Conv [55] with our strategy in Table 2 in the rebuttal PDF. We evaluate in the ScanNet benchmark as it is the most challenging in terms of generalization and our Conv [55] based method outperforms its baseline. 

### Can we recover from the baseline’s failures ?

Figure 2 in the rebuttal PDF shows cases where we can obtain good reconstructions for baseline failures, which shows that our strategy of insuring inputs are at the levelset is effective. 

### Additional Limitations 

We provide additional limitations in the rebuttal PDF and we will discuss them further in the next version. First ablation of number of nystrom samples m in Table 3 shows that the KRR can possibly suffer from overfitting for large values of m, hence the choice of this hyperparameter can be challenging. Additionally, Figure 2 show an examples where, despite improving on the reconstruction of the network baseline, it is still challenging for our method to recover the finest structures and intricate details of complex shapes. 

Meta-review: This paper was reviewed by five knowledgeable referees. Initial feedback indicated that the approach is technically sound (JKo3, 7xQW), novel (Pm8Q, q2Vh), and boasts high accuracy (6hDU, q2Vh, LSLk). While the paper initially received varied scores, some reviewers expressed concerns regarding the motivation and justification for the technical choices (6hDU, 7xQW), as well as its comparison to some recent Neural kernel-based reconstruction works (LSLk, 7xQW). The author addressed these concerns, prompting all reviewers to revise and some to increase their scores, eventually reaching a unanimous decision for acceptance. After thorough consideration of the feedback, the AC agrees with the reviewers and has decided to accept this paper, with a strong recommendation for the authors to integrate the suggested changes into the final manuscript.
Decision: Accept (poster)